Oct 30 18:32:25.025898 ip-10-0-142-153 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:32:25.507781 ip-10-0-142-153 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:32:25.507781 ip-10-0-142-153 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:32:25.507781 ip-10-0-142-153 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:32:25.507781 ip-10-0-142-153 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:32:25.512162 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.511954    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:25.512162 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512160    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:25.512162 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512167    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:25.512162 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512179    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512185    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512189    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512194    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512198    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512203    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512207    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512211    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512215    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512221    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512225    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512229    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512233    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512242    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512249    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512253    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512258    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512262    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512266    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512271    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:25.512421 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512276    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512280    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512284    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512288    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512293    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512302    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512307    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512311    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512317    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512322    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512326    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512330    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512336    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512341    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512347    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512352    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512369    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512376    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512386    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512390    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:25.513166 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512396    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512401    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512406    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512410    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512415    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512420    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512425    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512430    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512456    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512461    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512467    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512477    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512481    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512486    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512490    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512495    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512500    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512504    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512509    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:25.514081 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512514    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512518    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512523    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512527    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512536    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512541    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512548    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512557    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512566    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512571    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512577    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512581    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512586    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512604    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512609    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512615    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512625    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512632    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512636    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512642    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:25.514905 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512646    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512651    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512655    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512659    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512664    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512670    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512674    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512680    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512685    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512695    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512699    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512704    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512708    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512713    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512717    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512721    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512728    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512735    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512740    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:25.515737 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512746    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512752    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512762    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512767    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.512785    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514752    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514778    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514785    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514791    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514795    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514801    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514808    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514815    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514820    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514824    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514830    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514835    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514840    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514845    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:25.516552 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514850    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514854    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514858    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514863    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514868    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514872    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514877    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514882    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514887    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514891    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514896    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514901    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514906    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514910    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514915    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514922    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514928    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514934    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514940    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514946    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:25.517377 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514951    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514955    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514960    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514964    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514969    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514974    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514978    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514983    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514988    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514992    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.514996    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515001    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515005    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515009    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515013    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515018    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515025    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515029    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515034    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:25.518294 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515039    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515045    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515050    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515056    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515061    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515065    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515070    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515076    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515081    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515086    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515091    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515096    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515101    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515106    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515110    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515115    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515119    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515124    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515129    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:25.519207 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515134    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515139    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515143    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515148    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515155    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515160    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515166    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515171    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515176    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515180    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515185    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515190    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515195    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515198    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515204    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515209    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515214    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515219    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515223    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515228    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515233    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:25.519986 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515237    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515244    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515250    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515254    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515259    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515263    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515268    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515272    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515277    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515281    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515285    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515290    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.515295    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516363    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516385    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516393    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516402    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516410    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516415    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:25.520532 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516421    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516426    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516431    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516457    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516462    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516466    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516472    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516477    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516484    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516488    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516493    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516497    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516505    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516512    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516517    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516522    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516526    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516530    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516535    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:25.521194 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516539    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516543    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516547    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516551    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516555    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516559    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516563    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516567    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516574    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516579    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516586    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516590    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516595    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516599    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516603    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516608    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516612    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516616    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516621    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516626    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:25.521864 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516631    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516636    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516640    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516645    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516649    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516654    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516658    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516663    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516667    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516673    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516677    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516683    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516688    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516693    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516697    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516702    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516707    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516712    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516716    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:25.522378 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516721    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516725    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516730    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516735    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516750    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516755    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516761    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516766    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516771    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516776    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516780    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516785    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516791    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516795    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516800    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516807    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516812    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516817    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516822    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516826    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:25.522858 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516831    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516835    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516840    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516844    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516848    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516854    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516858    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516862    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516867    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516871    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516875    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516880    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516885    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516889    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516894    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516899    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516903    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516917    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516924    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516928    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:25.523633 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516932    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.516936    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517176    2581 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517191    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517211    2581 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517219    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517226    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517232    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517239    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517246    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517251    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517256    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517263    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517269    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517275    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517280    2581 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517284    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517289    2581 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517294    2581 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517298    2581 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517309    2581 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517314    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517319    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517324    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:32:25.524128 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517330    2581 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517337    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517343    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517349    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517354    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517359    2581 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517373    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517378    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517383    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517388    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517395    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517400    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517407    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517411    2581 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517416    2581 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517421    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517458    2581 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517465    2581 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517470    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517476    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517481    2581 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517495    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517501    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517506    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517512    2581 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:32:25.524776 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517517    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517523    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517527    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517532    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517537    2581 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517543    2581 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517548    2581 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517555    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517561    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517569    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517575    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517580    2581 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517585    2581 flags.go:64] FLAG: --help="false"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517591    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517606    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517611    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517616    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517622    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517629    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517634    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517639    2581 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517644    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517649    2581 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:32:25.525419 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517654    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517660    2581 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517665    2581 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517671    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517676    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517682    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517687    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517692    2581 flags.go:64] FLAG: --lock-file=""
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517697    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517701    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517708    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517720    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517725    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517733    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517738    2581 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517742    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517748    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517753    2581 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517760    2581 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517771    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517776    2581 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517783    2581 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517789    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517794    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517801    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:32:25.526035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517807    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517812    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517817    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517823    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517840    2581 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517846    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517852    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517857    2581 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517861    2581 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517866    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517872    2581 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517877    2581 flags.go:64] FLAG: --port="10250"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517882    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517888    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-01ff1c475c929cdac"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517893    2581 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517898    2581 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517904    2581 flags.go:64] FLAG: --register-node="true"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517909    2581 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517916    2581 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517921    2581 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517927    2581 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517932    2581 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517940    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517946    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517951    2581 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517959    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:32:25.526738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517965    2581 flags.go:64] FLAG: --runonce="false"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517971    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517976    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517982    2581 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517986    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517991    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.517998    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518004    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518010    2581 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518014    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518020    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518025    2581 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518029    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518034    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518039    2581 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518045    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518055    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518060    2581 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518065    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518077    2581 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518082    2581 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518087    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518104    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518110    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518116    2581 flags.go:64] FLAG: --v="2"
Oct 30 18:32:25.527497 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518125    2581 flags.go:64] FLAG: --version="false"
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518133    2581 flags.go:64] FLAG: --vmodule=""
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518145    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.518151    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518428    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518459    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518466    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518471    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518475    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518479    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518484    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518488    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518493    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518497    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518505    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518510    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518514    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518519    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518523    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518527    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:25.528136 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518532    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518536    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518540    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518544    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518548    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518552    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518556    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518560    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518564    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518568    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518573    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518577    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518581    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518587    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518603    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518610    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518617    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518626    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518637    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:25.528671 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518642    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518648    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518653    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518658    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518663    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518668    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518673    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518677    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518685    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518690    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518695    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518700    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518704    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518709    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518713    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518718    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518722    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518727    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518732    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518736    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:25.529150 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518740    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518744    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518749    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518753    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518757    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518761    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518767    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518774    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518779    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518785    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518789    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518796    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518800    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518804    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518810    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518814    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518820    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518824    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518828    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:25.529814 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518833    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518837    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518845    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518850    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518854    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518859    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518865    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518871    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518876    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518880    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518884    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518888    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518893    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518897    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518902    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518907    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518911    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518916    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518921    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:25.530382 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518926    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518932    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518936    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518941    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518946    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518953    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518958    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518963    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518967    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518972    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518976    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518980    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.518985    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.519053    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519129    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519138    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:25.530892 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519151    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519156    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519161    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519165    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519170    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519175    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519180    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519184    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519189    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519193    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519198    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519202    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519206    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519210    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519214    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519218    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519222    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519226    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519232    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519237    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:25.531289 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519241    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519247    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519251    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519256    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519261    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519265    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519269    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519274    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519278    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519286    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519293    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519298    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519304    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519311    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519316    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519321    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519325    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519330    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519334    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:25.531797 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519338    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519344    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519349    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519354    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519358    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519364    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519368    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519373    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519378    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519382    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519387    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519394    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519399    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519402    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519407    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519411    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519415    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519420    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519424    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519428    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:25.532275 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519452    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519458    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519462    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519467    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519471    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519475    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519480    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519487    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519492    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519497    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519501    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519506    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519512    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519516    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519521    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519526    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519531    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519539    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519545    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:25.532844 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519551    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519557    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519561    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519566    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519574    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519579    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519583    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519588    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519592    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519597    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519601    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519606    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519610    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519615    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519619    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519624    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519628    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519632    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519636    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519641    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:25.533321 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519645    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519651    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519656    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519661    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519667    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.519672    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.529122    2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.529155    2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.530064    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.530077    2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530176    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530185    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530188    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530193    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530198    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530201    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530205    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530208    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530211    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:25.533875 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530214    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530217    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530219    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530222    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530225    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530229    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530232    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530235    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530238    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530240    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530243    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530246    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530248    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530251    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530253    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530256    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530258    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530261    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530264    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530267    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:25.534358 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530272    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530275    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530278    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530281    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530284    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530287    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530290    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530293    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530295    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530298    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530301    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530304    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530307    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530309    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530312    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530315    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530317    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530320    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530323    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530326    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:25.534944 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530329    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530331    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530336    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530340    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530343    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530345    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530348    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530351    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530354    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530357    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530360    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530362    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530366    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530369    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530372    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530374    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530377    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530380    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530382    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:25.536256 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530385    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530388    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530391    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530394    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530397    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530399    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530402    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530405    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530408    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530411    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530414    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530417    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530420    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530422    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530426    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530429    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530453    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530457    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530460    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530463    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:25.536752 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530466    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530468    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530471    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530474    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530477    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530480    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530483    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530487    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530490    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530493    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530496    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530499    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530502    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530504    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530508    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530510    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530513    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:25.537265 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530516    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.530568    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530764    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530772    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530776    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530779    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530781    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530784    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530787    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530790    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530793    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530795    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530798    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530801    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530804    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530806    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:25.537852 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530810    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530813    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530815    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530818    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530821    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530824    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530826    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530836    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530840    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530842    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530844    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530847    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530851    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530855    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530858    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530862    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530864    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530867    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530870    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:25.538266 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530873    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530876    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530879    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530881    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530884    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530887    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530890    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530893    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530896    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530899    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530902    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530905    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530908    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530911    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530914    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530916    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530919    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530921    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530924    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530926    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:25.538766 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530929    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530939    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530941    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530945    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530947    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530950    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530952    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530955    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530957    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530959    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530962    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530964    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530967    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530970    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530972    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530975    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530977    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530980    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530983    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530986    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:25.539250 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530988    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530991    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.530994    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531004    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531007    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531010    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531013    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531016    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531019    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531021    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531024    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531027    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531030    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531033    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531041    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531044    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531047    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531050    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531052    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:25.539762 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531055    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531058    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531060    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531062    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531067    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531069    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531072    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531074    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531078    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531082    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531085    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531089    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531093    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531097    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.531125    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531170    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:32:25.540240 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531175    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531178    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531181    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531184    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531187    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531190    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531192    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531195    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531198    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531200    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531203    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531205    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531215    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531218    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531221    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531223    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531226    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531229    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531231    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:32:25.540952 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531234    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531236    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531239    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531241    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531244    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531247    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531275    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531279    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531284    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531287    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531291    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531294    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531297    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531302    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531306    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531309    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531312    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531314    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531317    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531320    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:32:25.541431 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531322    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531325    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531327    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531330    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531332    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531335    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531344    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531347    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531354    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531356    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531359    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531362    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531365    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531368    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531372    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531375    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531378    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531381    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531383    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:32:25.542125 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531386    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531389    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531391    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531394    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531396    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531400    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531404    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531407    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531410    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531412    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531415    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531417    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531420    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531423    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531425    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531428    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531431    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531452    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531456    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:32:25.542724 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531459    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531469    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531474    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531477    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531480    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531483    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531486    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531489    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531498    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531501    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531504    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531507    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531510    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531513    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531516    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531519    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531522    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531525    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531527    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531530    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:32:25.543238 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531533    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:32:25.543796 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531535    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:32:25.543796 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531538    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:32:25.543796 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531541    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:32:25.543796 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531543    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:32:25.543796 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531549    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:32:25.543796 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531552    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:32:25.543796 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:25.531555    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:32:25.543796 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.531647    2581 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:32:25.543796 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.534624    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 18:32:25.543796 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.535854    2581 server.go:1014] "Starting client certificate rotation"
Oct 30 18:32:25.543796 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.535970    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:32:25.543796 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.536033    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:32:25.543796 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.538724    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:32:25.543796 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.540261    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:32:25.557189 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.557154    2581 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:32:25.562039 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.562018    2581 log.go:25] "Validated CRI v1 image API"
Oct 30 18:32:25.563024 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.562989    2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:32:25.566319 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.566277    2581 fs.go:135] Filesystem UUIDs: map[3ba165b1-f104-4863-a297-4f1404590f8e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e26d2272-0779-43f8-b926-34abf9f0af0c:/dev/nvme0n1p4]
Oct 30 18:32:25.566319 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.566310    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 18:32:25.574025 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.573839    2581 manager.go:217] Machine: {Timestamp:2026-10-30 18:32:25.572126232 +0000 UTC m=+0.406514106 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100118 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec292fbae9a68021022edcbd4c3e98b1 SystemUUID:ec292fba-e9a6-8021-022e-dcbd4c3e98b1 BootID:709cbf4e-a181-4fb9-9b55-68c21fd38de8 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e9:a9:55:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e9:a9:55:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:44:0f:be:0d:60 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:32:25.574025 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.573992    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:32:25.574214 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.574183    2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:32:25.578047 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.578000    2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:32:25.578260 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.578051    2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-153.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:32:25.578311 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.578269    2581 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:32:25.578311 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.578280    2581 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:32:25.578373 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.578320    2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:32:25.578373 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.578338    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:32:25.579660 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.579632    2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:32:25.584538 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.584508    2581 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:32:25.584739 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.584728    2581 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:32:25.584784 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.584776    2581 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:32:25.584812 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.584792    2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:32:25.584812 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.584805    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:32:25.584892 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.584815    2581 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:32:25.584892 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.584826    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:32:25.585336 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.585322    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:32:25.588127 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.588093    2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:32:25.589297 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.589268    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:32:25.595383 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.595349    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:32:25.596358 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.596343    2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:32:25.596429 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.596376    2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:32:25.596633 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.596621    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:32:25.596665 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.596639    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:32:25.596665 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.596647    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:32:25.596665 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.596653    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:32:25.596665 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.596659    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:32:25.596665 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.596665    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:32:25.597303 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.597292    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:32:25.597303 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.597303    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:32:25.597362 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.597312    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:32:25.597362 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.597319    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:32:25.597362 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.597331    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:32:25.597362 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.597343    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:32:25.598983 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.598958    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:32:25.599094 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.598991    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:32:25.602275 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.602238    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-153.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:32:25.603982 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.603958    2581 server.go:1263] "Started kubelet"
Oct 30 18:32:25.604244 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.604043    2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:32:25.605012 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.604920    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:32:25.605245 ip-10-0-142-153 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:32:25.608373 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.608316    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:32:25.610120 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.610080    2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:32:25.610703 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.610644    2581 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:32:25.610844 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.610823    2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 18:32:25.613981 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.613954    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rfbdk"
Oct 30 18:32:25.619518 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.619479    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:32:25.620173 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.620135    2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:32:25.620520 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.620490    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-153.us-east-2.compute.internal\" not found"
Oct 30 18:32:25.620969 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.620949    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:32:25.620969 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.620948    2581 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:32:25.621110 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.620981    2581 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:32:25.621110 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.621076    2581 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:32:25.621110 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.621087    2581 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:32:25.621388 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.621367    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:32:25.621388 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.621387    2581 factory.go:55] Registering systemd factory
Oct 30 18:32:25.621554 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.621396    2581 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:32:25.621658 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.621645    2581 factory.go:153] Registering CRI-O factory
Oct 30 18:32:25.621721 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.621662    2581 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:32:25.621721 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.621679    2581 factory.go:103] Registering Raw factory
Oct 30 18:32:25.621721 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.621691    2581 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:32:25.622846 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.622807    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 18:32:25.623021 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.623007    2581 manager.go:319] Starting recovery of all containers
Oct 30 18:32:25.624022 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.622730    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-153.us-east-2.compute.internal.18e36249c52ef532  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-153.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-153.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:32:25.60391301 +0000 UTC m=+0.438300870,LastTimestamp:2026-10-30 18:32:25.60391301 +0000 UTC m=+0.438300870,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-153.us-east-2.compute.internal,}"
Oct 30 18:32:25.628035 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.627991    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-153.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 18:32:25.632014 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.631982    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 18:32:25.633812 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.633788    2581 manager.go:324] Recovery completed
Oct 30 18:32:25.636993 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.636961    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-153.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 18:32:25.639198 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.639176    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:32:25.639686 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.639669    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rfbdk"
Oct 30 18:32:25.641542 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.641521    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:32:25.641635 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.641558    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:32:25.641635 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.641569    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:32:25.642276 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.642261    2581 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:32:25.642276 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.642271    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:32:25.642413 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.642274    2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:32:25.642413 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.642346    2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:32:25.643178 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.643163    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:32:25.643254 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.643191    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:32:25.643254 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.643204    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:32:25.644753 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.644734    2581 policy_none.go:50] "Start"
Oct 30 18:32:25.644829 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.644763    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:32:25.644829 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.644785    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:32:25.645350 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.645263    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-153.us-east-2.compute.internal.18e36249c76d21aa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-153.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-153.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-153.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:32:25.641542058 +0000 UTC m=+0.475929919,LastTimestamp:2026-10-30 18:32:25.641542058 +0000 UTC m=+0.475929919,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-153.us-east-2.compute.internal,}"
Oct 30 18:32:25.647007 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.646990    2581 policy_none.go:44] "Start"
Oct 30 18:32:25.707145 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.689241    2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:32:25.707145 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.695978    2581 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:32:25.707145 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.696660    2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:32:25.707145 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.696676    2581 server.go:87] "Starting device plugin registration server"
Oct 30 18:32:25.707145 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.696954    2581 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:32:25.707145 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.696967    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:32:25.707145 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.697086    2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:32:25.707145 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.697211    2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:32:25.707145 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.697220    2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:32:25.707145 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.697880    2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:32:25.707145 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.697940    2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-153.us-east-2.compute.internal\" not found"
Oct 30 18:32:25.711203 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.711158    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:32:25.712854 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.712821    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:32:25.712854 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.712857    2581 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:32:25.713081 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.712891    2581 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:32:25.713081 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.712936    2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 18:32:25.719055 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.719012    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:32:25.798078 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.797981    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:32:25.799473 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.799427    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:32:25.799614 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.799489    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:32:25.799614 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.799501    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:32:25.799614 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.799531    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:25.810546 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.810513    2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:25.810653 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.810555    2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-153.us-east-2.compute.internal\": node \"ip-10-0-142-153.us-east-2.compute.internal\" not found"
Oct 30 18:32:25.813579 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.813548    2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal"]
Oct 30 18:32:25.813718 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.813629    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:32:25.814889 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.814850    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:32:25.815034 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.814896    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:32:25.815034 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.814913    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:32:25.817470 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.817430    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:32:25.817878 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.817609    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:25.817878 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.817670    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:32:25.818405 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.818372    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:32:25.818522 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.818412    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:32:25.818522 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.818372    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:32:25.818522 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.818477    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:32:25.818522 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.818490    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:32:25.818649 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.818430    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:32:25.820802 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.820783    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:25.820886 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.820828    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:32:25.821977 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.821947    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:32:25.822074 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.821989    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:32:25.822074 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.821994    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b46186b28ca7c09f5723c9f8bbbafe8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"b46186b28ca7c09f5723c9f8bbbafe8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:25.822074 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.822006    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:32:25.833802 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.833772    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-153.us-east-2.compute.internal\" not found"
Oct 30 18:32:25.841572 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.841545    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-153.us-east-2.compute.internal\" not found" node="ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:25.843698 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.843673    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-153.us-east-2.compute.internal\" not found" node="ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:25.923047 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.923004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b46186b28ca7c09f5723c9f8bbbafe8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"b46186b28ca7c09f5723c9f8bbbafe8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:25.923047 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.923057    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b46186b28ca7c09f5723c9f8bbbafe8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"b46186b28ca7c09f5723c9f8bbbafe8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:25.923312 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.923081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c870f7ea63961faf808ec834ef3f426f-config\") pod \"kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"c870f7ea63961faf808ec834ef3f426f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:25.923312 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:25.923147    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b46186b28ca7c09f5723c9f8bbbafe8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"b46186b28ca7c09f5723c9f8bbbafe8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:25.934149 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:25.934112    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-153.us-east-2.compute.internal\" not found"
Oct 30 18:32:26.023742 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:26.023699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b46186b28ca7c09f5723c9f8bbbafe8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"b46186b28ca7c09f5723c9f8bbbafe8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:26.023742 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:26.023746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c870f7ea63961faf808ec834ef3f426f-config\") pod \"kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"c870f7ea63961faf808ec834ef3f426f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:26.023978 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:26.023819    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c870f7ea63961faf808ec834ef3f426f-config\") pod \"kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"c870f7ea63961faf808ec834ef3f426f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:26.023978 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:26.023820    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b46186b28ca7c09f5723c9f8bbbafe8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"b46186b28ca7c09f5723c9f8bbbafe8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:26.034919 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:26.034846    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-153.us-east-2.compute.internal\" not found"
Oct 30 18:32:26.135943 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:26.135832    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-153.us-east-2.compute.internal\" not found"
Oct 30 18:32:26.147105 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:26.147057    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:26.149261 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:26.149234    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:27.588853 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:27.588814    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-153.us-east-2.compute.internal\" not found"
Oct 30 18:32:27.689480 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:27.689415    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-153.us-east-2.compute.internal\" not found"
Oct 30 18:32:27.790090 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:27.789978    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-153.us-east-2.compute.internal\" not found"
Oct 30 18:32:27.860329 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.860288    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:32:27.873262 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.873225    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:27.887483 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.887442    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:32:27.888389 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.888367    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 18:32:27.888560 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.888528    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:32:27.888608 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.888561    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:32:27.889835 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.889812    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:32:27.916821 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.916776    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:32:27.937986 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.937945    2581 apiserver.go:52] "Watching apiserver"
Oct 30 18:32:27.948220 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.948174    2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:32:27.948594 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.948569    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-8vlwv","openshift-multus/network-metrics-daemon-ngsb2","openshift-image-registry/node-ca-9mqb5","openshift-multus/multus-5nnbq","openshift-network-diagnostics/network-check-target-n52lm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl","openshift-dns/node-resolver-h64hp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-vlnz5","kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-z4csl","openshift-network-operator/iptables-alerter-h7gnp","openshift-ovn-kubernetes/ovnkube-node-2pz2f","kube-system/global-pull-secret-syncer-qmbfz"]
Oct 30 18:32:27.951244 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.951211    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:32:27.953594 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.953561    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:27.953737 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:27.953650    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngsb2" podUID="48282811-970d-41fb-8745-54023a7558d3"
Oct 30 18:32:27.955738 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.955713    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:32:27.956887 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.956863    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:32:27.956887 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.956881    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:32:27.957009 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.956947    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fshmx\""
Oct 30 18:32:27.959514 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.959485    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.961259 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.961230    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:27.961387 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:27.961349    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n52lm" podUID="91317f6c-d730-44f1-a5a1-96ba9c1a6aec"
Oct 30 18:32:27.963427 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.963400    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:27.963944 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.963914    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:32:27.966457 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.966435    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:32:27.968887 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.968855    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:27.972360 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.972318    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 18:32:27.973056 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.973030    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:27.975877 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.975844    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:32:27.978758 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.978730    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:27.979548 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.979523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41efb608-1c74-4ab4-9693-53495bfc26b9-konnectivity-ca\") pod \"konnectivity-agent-8vlwv\" (UID: \"41efb608-1c74-4ab4-9693-53495bfc26b9\") " pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:32:27.979670 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.979555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-socket-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:27.979670 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.979574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-device-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:27.979670 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.979592    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-run\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:27.979808 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.979665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/701c1b5b-7bf7-4f11-9623-f50f2181c384-iptables-alerter-script\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:32:27.979808 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.979710    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c151e21-28d5-4d4a-8b95-b92d623582c5-tmp-dir\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:32:27.979808 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.979753    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-var-lib-kubelet\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:27.979921 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.979800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-host\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:27.979921 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.979835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-tuned\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:27.979921 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.979879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jh8p\" (UniqueName: \"kubernetes.io/projected/24897d0a-ab73-4e75-ac67-c5f5103571d0-kube-api-access-4jh8p\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:27.979921 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.979905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-cni-multus\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.980079 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.979927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:27.980079 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.979947    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-multus-certs\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.980079 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.979966    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:27.980079 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-kubernetes\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:27.980079 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980050    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-lib-modules\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:27.980249 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-netns\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.980249 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:27.980249 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp7fk\" (UniqueName: \"kubernetes.io/projected/382d4192-0bfc-4626-820d-5cf183296180-kube-api-access-tp7fk\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:32:27.980249 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btss5\" (UniqueName: \"kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5\") pod \"network-check-target-n52lm\" (UID: \"91317f6c-d730-44f1-a5a1-96ba9c1a6aec\") " pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:27.980395 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-etc-selinux\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:27.980395 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysconfig\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:27.980395 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-cnibin\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.980594 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-hostroot\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.980594 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/884761af-7ea5-4be1-bd94-e736df037d21-multus-daemon-config\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.980594 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c151e21-28d5-4d4a-8b95-b92d623582c5-hosts-file\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:32:27.980594 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980536    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-cnibin\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:27.980594 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980577    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-systemd\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:27.980740 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24897d0a-ab73-4e75-ac67-c5f5103571d0-tmp\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:27.980740 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980654    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-registration-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:27.980740 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-sys-fs\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:27.980740 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:27.980854 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjz6c\" (UniqueName: \"kubernetes.io/projected/701c1b5b-7bf7-4f11-9623-f50f2181c384-kube-api-access-pjz6c\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:32:27.980854 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:27.980854 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ljt9\" (UniqueName: \"kubernetes.io/projected/9c151e21-28d5-4d4a-8b95-b92d623582c5-kube-api-access-5ljt9\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:32:27.980937 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/382d4192-0bfc-4626-820d-5cf183296180-host\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:32:27.980937 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-socket-dir-parent\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.980937 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-kubelet\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.981028 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxhdz\" (UniqueName: \"kubernetes.io/projected/ba289df1-672b-4bf2-85f6-eaffd2269631-kube-api-access-wxhdz\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:27.981028 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.980965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/382d4192-0bfc-4626-820d-5cf183296180-serviceca\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:32:27.981086 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981053    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/701c1b5b-7bf7-4f11-9623-f50f2181c384-host-slash\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:32:27.981138 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-cni-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.981177 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwpb5\" (UniqueName: \"kubernetes.io/projected/dd918ffa-ead9-4697-9045-da66e9c7f136-kube-api-access-mwpb5\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:27.981207 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-modprobe-d\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:27.981207 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysctl-d\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:27.981266 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-system-cni-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.981266 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-k8s-cni-cncf-io\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.981266 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981255    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:27.981353 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-system-cni-dir\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:27.981353 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-conf-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.981353 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-etc-kubernetes\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.981353 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysctl-conf\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:27.981462 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41efb608-1c74-4ab4-9693-53495bfc26b9-agent-certs\") pod \"konnectivity-agent-8vlwv\" (UID: \"41efb608-1c74-4ab4-9693-53495bfc26b9\") " pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:32:27.981462 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-cni-bin\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.981462 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt8v8\" (UniqueName: \"kubernetes.io/projected/48282811-970d-41fb-8745-54023a7558d3-kube-api-access-xt8v8\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:27.981462 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-sys\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:27.981462 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/884761af-7ea5-4be1-bd94-e736df037d21-cni-binary-copy\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.981636 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.981514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq592\" (UniqueName: \"kubernetes.io/projected/884761af-7ea5-4be1-bd94-e736df037d21-kube-api-access-vq592\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:27.982509 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.982488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:27.982605 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:27.982582    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbfz" podUID="faeb3114-7d5c-4d94-a00c-7af6f1c175df"
Oct 30 18:32:27.999081 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.999013    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:27:26 +0000 UTC" deadline="2028-04-23 02:53:01.000658114 +0000 UTC"
Oct 30 18:32:27.999081 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:27.999075    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12968h20m33.001585551s"
Oct 30 18:32:28.015084 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.015034    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:32:28.015332 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.015116    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kqw4m"
Oct 30 18:32:28.027228 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.027198    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kqw4m"
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065212    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065235    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065304    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065324    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s26z4\""
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065327    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065308    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065360    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065381    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065368    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-82xdm\""
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065401    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065415    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065419    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065404    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g44rk\""
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065463    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065473    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:32:28.065524 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065486    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:32:28.066439 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065620    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:32:28.066439 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065633    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:32:28.066439 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065663    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:32:28.066439 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065678    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jhsz4\""
Oct 30 18:32:28.066439 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065759    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:32:28.066439 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.065818    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:32:28.066439 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.066031    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9fr66\""
Oct 30 18:32:28.066439 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.066079    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:32:28.066439 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.066115    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:32:28.066439 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.066252    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qwdqr\""
Oct 30 18:32:28.066439 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.066292    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:32:28.066439 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.066350    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:32:28.066439 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.066400    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lc92h\""
Oct 30 18:32:28.067779 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.067688    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:32:28.068351 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.068328    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:32:28.068483 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.068406    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-59qb5\""
Oct 30 18:32:28.075757 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.075313    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:32:28.081716 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.081691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-env-overrides\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.081859 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.081729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-system-cni-dir\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:28.081859 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.081758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-conf-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.081974 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.081865    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-conf-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.081974 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.081881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-etc-kubernetes\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.081974 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.081924    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-system-cni-dir\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:28.081974 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.081929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-systemd-units\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.081974 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.081954    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-etc-kubernetes\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.082200 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.081979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:28.082200 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysctl-conf\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.082200 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-systemd\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.082200 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.082200 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.082200 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-node-log\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.082467 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-log-socket\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.082467 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082250    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysctl-conf\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.082467 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082259    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovnkube-config\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.082467 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41efb608-1c74-4ab4-9693-53495bfc26b9-agent-certs\") pod \"konnectivity-agent-8vlwv\" (UID: \"41efb608-1c74-4ab4-9693-53495bfc26b9\") " pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:32:28.082467 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-cni-bin\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.082467 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-etc-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.082467 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082443    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-cni-bin\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.082816 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082479    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-cni-netd\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.082816 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.082816 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt8v8\" (UniqueName: \"kubernetes.io/projected/48282811-970d-41fb-8745-54023a7558d3-kube-api-access-xt8v8\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:28.082816 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-sys\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.082816 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082691    2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:32:28.082816 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082764    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-sys\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.082816 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/884761af-7ea5-4be1-bd94-e736df037d21-cni-binary-copy\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.083190 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq592\" (UniqueName: \"kubernetes.io/projected/884761af-7ea5-4be1-bd94-e736df037d21-kube-api-access-vq592\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.083190 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-kubelet\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.083190 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-slash\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.083190 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41efb608-1c74-4ab4-9693-53495bfc26b9-konnectivity-ca\") pod \"konnectivity-agent-8vlwv\" (UID: \"41efb608-1c74-4ab4-9693-53495bfc26b9\") " pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:32:28.083190 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.082997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-socket-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:28.083190 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-device-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:28.083190 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-run\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.083190 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/701c1b5b-7bf7-4f11-9623-f50f2181c384-iptables-alerter-script\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:32:28.083190 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c151e21-28d5-4d4a-8b95-b92d623582c5-tmp-dir\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:32:28.083190 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-var-lib-kubelet\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.083190 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083177    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-socket-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:28.083704 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-host\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.083704 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-tuned\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.083704 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083229    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-host\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.083704 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jh8p\" (UniqueName: \"kubernetes.io/projected/24897d0a-ab73-4e75-ac67-c5f5103571d0-kube-api-access-4jh8p\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.083704 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083282    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-cni-multus\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.083704 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083348    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-var-lib-kubelet\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.083704 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083374    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-cni-multus\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.083704 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083457    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-run\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.084047 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083878    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/884761af-7ea5-4be1-bd94-e736df037d21-cni-binary-copy\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.084047 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083944    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/701c1b5b-7bf7-4f11-9623-f50f2181c384-iptables-alerter-script\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:32:28.084047 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083945    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-device-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:28.084047 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.083986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:28.084047 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-multus-certs\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.084047 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:28.084379 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084054    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41efb608-1c74-4ab4-9693-53495bfc26b9-konnectivity-ca\") pod \"konnectivity-agent-8vlwv\" (UID: \"41efb608-1c74-4ab4-9693-53495bfc26b9\") " pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:32:28.084379 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-kubernetes\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.084379 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084074    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-multus-certs\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.084379 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084088    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-kubernetes\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.084379 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-lib-modules\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.084379 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-netns\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.084379 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084171    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c151e21-28d5-4d4a-8b95-b92d623582c5-tmp-dir\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:32:28.084379 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:28.084379 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.084208    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:28.084379 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovn-node-metrics-cert\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.084379 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084218    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-lib-modules\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.084379 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084259    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-netns\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.084379 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/faeb3114-7d5c-4d94-a00c-7af6f1c175df-kubelet-config\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:28.084379 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.084332    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs podName:48282811-970d-41fb-8745-54023a7558d3 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:28.584302882 +0000 UTC m=+2.066002412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs") pod "network-metrics-daemon-ngsb2" (UID: "48282811-970d-41fb-8745-54023a7558d3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:28.084379 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084334    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084403    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp7fk\" (UniqueName: \"kubernetes.io/projected/382d4192-0bfc-4626-820d-5cf183296180-kube-api-access-tp7fk\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btss5\" (UniqueName: \"kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5\") pod \"network-check-target-n52lm\" (UID: \"91317f6c-d730-44f1-a5a1-96ba9c1a6aec\") " pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-etc-selinux\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysconfig\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-cnibin\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-hostroot\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/884761af-7ea5-4be1-bd94-e736df037d21-multus-daemon-config\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c151e21-28d5-4d4a-8b95-b92d623582c5-hosts-file\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-cnibin\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-systemd\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24897d0a-ab73-4e75-ac67-c5f5103571d0-tmp\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/faeb3114-7d5c-4d94-a00c-7af6f1c175df-dbus\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-registration-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084768    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-etc-selinux\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084778    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084801    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9sxz\" (UniqueName: \"kubernetes.io/projected/3f0072d4-229e-452e-a533-7bf54e6f5ba4-kube-api-access-w9sxz\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.085072 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-sys-fs\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-run-netns\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084855    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-cnibin\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovnkube-script-lib\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084956    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-sys-fs\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084963    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysconfig\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.084995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjz6c\" (UniqueName: \"kubernetes.io/projected/701c1b5b-7bf7-4f11-9623-f50f2181c384-kube-api-access-pjz6c\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085052    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-ovn\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085055    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085058    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-registration-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-cni-bin\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085087    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ljt9\" (UniqueName: \"kubernetes.io/projected/9c151e21-28d5-4d4a-8b95-b92d623582c5-kube-api-access-5ljt9\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085088    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-hostroot\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085106    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c151e21-28d5-4d4a-8b95-b92d623582c5-hosts-file\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085167    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-systemd\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.085945 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/382d4192-0bfc-4626-820d-5cf183296180-host\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-socket-dir-parent\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-kubelet\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxhdz\" (UniqueName: \"kubernetes.io/projected/ba289df1-672b-4bf2-85f6-eaffd2269631-kube-api-access-wxhdz\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/382d4192-0bfc-4626-820d-5cf183296180-serviceca\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085404    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-socket-dir-parent\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/701c1b5b-7bf7-4f11-9623-f50f2181c384-host-slash\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085443    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/701c1b5b-7bf7-4f11-9623-f50f2181c384-host-slash\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-cni-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwpb5\" (UniqueName: \"kubernetes.io/projected/dd918ffa-ead9-4697-9045-da66e9c7f136-kube-api-access-mwpb5\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085483    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/884761af-7ea5-4be1-bd94-e736df037d21-multus-daemon-config\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085539    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-cnibin\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-modprobe-d\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085616    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/382d4192-0bfc-4626-820d-5cf183296180-host\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysctl-d\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085627    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-system-cni-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.086629 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-k8s-cni-cncf-io\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.087106 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085773    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-cni-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.087106 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:28.087106 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085842    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-k8s-cni-cncf-io\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.087106 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085850    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-var-lib-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.087106 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085899    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysctl-d\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.087106 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085895    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-system-cni-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.087106 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.085960    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-kubelet\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.087106 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.086012    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-modprobe-d\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.087106 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.086401    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/382d4192-0bfc-4626-820d-5cf183296180-serviceca\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:32:28.087106 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.086701    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:28.087779 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.087760    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41efb608-1c74-4ab4-9693-53495bfc26b9-agent-certs\") pod \"konnectivity-agent-8vlwv\" (UID: \"41efb608-1c74-4ab4-9693-53495bfc26b9\") " pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:32:28.088765 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.088745    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24897d0a-ab73-4e75-ac67-c5f5103571d0-tmp\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.088846 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.088800    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-tuned\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.097582 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.097555    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:32:28.097582 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.097585    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:32:28.097816 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.097597    2581 projected.go:196] Error preparing data for projected volume kube-api-access-btss5 for pod openshift-network-diagnostics/network-check-target-n52lm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:28.097816 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.097672    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5 podName:91317f6c-d730-44f1-a5a1-96ba9c1a6aec nodeName:}" failed. No retries permitted until 2026-10-30 18:32:28.59765112 +0000 UTC m=+2.079350674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-btss5" (UniqueName: "kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5") pod "network-check-target-n52lm" (UID: "91317f6c-d730-44f1-a5a1-96ba9c1a6aec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:28.100495 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.100459    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ljt9\" (UniqueName: \"kubernetes.io/projected/9c151e21-28d5-4d4a-8b95-b92d623582c5-kube-api-access-5ljt9\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:32:28.100658 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.100509    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp7fk\" (UniqueName: \"kubernetes.io/projected/382d4192-0bfc-4626-820d-5cf183296180-kube-api-access-tp7fk\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:32:28.100749 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.100730    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jh8p\" (UniqueName: \"kubernetes.io/projected/24897d0a-ab73-4e75-ac67-c5f5103571d0-kube-api-access-4jh8p\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.101692 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.101674    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq592\" (UniqueName: \"kubernetes.io/projected/884761af-7ea5-4be1-bd94-e736df037d21-kube-api-access-vq592\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.101746 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.101699    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt8v8\" (UniqueName: \"kubernetes.io/projected/48282811-970d-41fb-8745-54023a7558d3-kube-api-access-xt8v8\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:28.106763 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.106728    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxhdz\" (UniqueName: \"kubernetes.io/projected/ba289df1-672b-4bf2-85f6-eaffd2269631-kube-api-access-wxhdz\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:28.109697 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.109667    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjz6c\" (UniqueName: \"kubernetes.io/projected/701c1b5b-7bf7-4f11-9623-f50f2181c384-kube-api-access-pjz6c\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:32:28.109697 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.109687    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwpb5\" (UniqueName: \"kubernetes.io/projected/dd918ffa-ead9-4697-9045-da66e9c7f136-kube-api-access-mwpb5\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:28.116553 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.116522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:32:28.130447 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.130414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:32:28.186286 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-var-lib-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.186286 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-env-overrides\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.186566 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-systemd-units\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.186566 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:28.186566 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186369    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-var-lib-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.186566 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-systemd\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.186566 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.186566 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186440    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-systemd-units\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.186566 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.186566 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.186486    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:28.186566 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186510    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.186566 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-node-log\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.186566 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.186556    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret podName:faeb3114-7d5c-4d94-a00c-7af6f1c175df nodeName:}" failed. No retries permitted until 2026-10-30 18:32:28.686529222 +0000 UTC m=+2.168228781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret") pod "global-pull-secret-syncer-qmbfz" (UID: "faeb3114-7d5c-4d94-a00c-7af6f1c175df") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:28.186566 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186559    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-node-log\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.186566 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186556    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186612    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-systemd\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-log-socket\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186645    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-log-socket\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovnkube-config\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-etc-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-cni-netd\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186747    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-etc-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186805    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-cni-netd\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-kubelet\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186839    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-kubelet\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186810    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186848    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-env-overrides\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-slash\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186886    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-slash\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.186955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovn-node-metrics-cert\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187102 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.187013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/faeb3114-7d5c-4d94-a00c-7af6f1c175df-kubelet-config\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:28.187810 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.187071    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/faeb3114-7d5c-4d94-a00c-7af6f1c175df-kubelet-config\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:28.187810 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.187149    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovnkube-config\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187810 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.187173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/faeb3114-7d5c-4d94-a00c-7af6f1c175df-dbus\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:28.187810 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.187198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9sxz\" (UniqueName: \"kubernetes.io/projected/3f0072d4-229e-452e-a533-7bf54e6f5ba4-kube-api-access-w9sxz\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187810 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.187227    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-run-netns\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187810 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.187236    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/faeb3114-7d5c-4d94-a00c-7af6f1c175df-dbus\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:28.187810 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.187257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovnkube-script-lib\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187810 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.187276    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-run-netns\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187810 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.187364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-ovn\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187810 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.187392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-cni-bin\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187810 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.187483    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-ovn\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187810 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.187496    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-cni-bin\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.187810 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.187758    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovnkube-script-lib\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.190291 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.190262    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovn-node-metrics-cert\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.202193 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.202161    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9sxz\" (UniqueName: \"kubernetes.io/projected/3f0072d4-229e-452e-a533-7bf54e6f5ba4-kube-api-access-w9sxz\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.212212 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:28.212158    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46186b28ca7c09f5723c9f8bbbafe8c.slice/crio-d2edb2b019e72eea9d832b421121189a8b4a30a8107ddeca4cb7a3a7726b401f WatchSource:0}: Error finding container d2edb2b019e72eea9d832b421121189a8b4a30a8107ddeca4cb7a3a7726b401f: Status 404 returned error can't find the container with id d2edb2b019e72eea9d832b421121189a8b4a30a8107ddeca4cb7a3a7726b401f
Oct 30 18:32:28.214971 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:28.214935    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc870f7ea63961faf808ec834ef3f426f.slice/crio-6eccfeda5b686bab80f177d0880cb511c20ed2ceae94660875b10d7c7846a66c WatchSource:0}: Error finding container 6eccfeda5b686bab80f177d0880cb511c20ed2ceae94660875b10d7c7846a66c: Status 404 returned error can't find the container with id 6eccfeda5b686bab80f177d0880cb511c20ed2ceae94660875b10d7c7846a66c
Oct 30 18:32:28.219845 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.219823    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:32:28.280805 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.280732    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:32:28.292509 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:28.292413    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41efb608_1c74_4ab4_9693_53495bfc26b9.slice/crio-a9f27f3523180933c4e6aa107ff07ce589d7d707440f9b74848dff1fe3249002 WatchSource:0}: Error finding container a9f27f3523180933c4e6aa107ff07ce589d7d707440f9b74848dff1fe3249002: Status 404 returned error can't find the container with id a9f27f3523180933c4e6aa107ff07ce589d7d707440f9b74848dff1fe3249002
Oct 30 18:32:28.307383 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.307335    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:32:28.317655 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:28.317619    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382d4192_0bfc_4626_820d_5cf183296180.slice/crio-923d622a4a41829901a91e68bcb94753674b1c2685d212f58f13cb0c53c44828 WatchSource:0}: Error finding container 923d622a4a41829901a91e68bcb94753674b1c2685d212f58f13cb0c53c44828: Status 404 returned error can't find the container with id 923d622a4a41829901a91e68bcb94753674b1c2685d212f58f13cb0c53c44828
Oct 30 18:32:28.326632 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.326584    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5nnbq"
Oct 30 18:32:28.337222 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:28.337181    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884761af_7ea5_4be1_bd94_e736df037d21.slice/crio-e4ac273964b9f14facbe2259b965282a07321cf58f2e353bcdad262a4f216902 WatchSource:0}: Error finding container e4ac273964b9f14facbe2259b965282a07321cf58f2e353bcdad262a4f216902: Status 404 returned error can't find the container with id e4ac273964b9f14facbe2259b965282a07321cf58f2e353bcdad262a4f216902
Oct 30 18:32:28.354763 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.354718    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:32:28.359091 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.359068    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:32:28.367143 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:28.367089    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd918ffa_ead9_4697_9045_da66e9c7f136.slice/crio-5b07f85050d7d978a0a56fd257596e801ed11cfce1425a5f08d52e1c44e9b03e WatchSource:0}: Error finding container 5b07f85050d7d978a0a56fd257596e801ed11cfce1425a5f08d52e1c44e9b03e: Status 404 returned error can't find the container with id 5b07f85050d7d978a0a56fd257596e801ed11cfce1425a5f08d52e1c44e9b03e
Oct 30 18:32:28.370715 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:28.370689    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c151e21_28d5_4d4a_8b95_b92d623582c5.slice/crio-f710dd87a10bcb0045792e084e350e9dd3a42f64ed1fcef9696675df3a5c953a WatchSource:0}: Error finding container f710dd87a10bcb0045792e084e350e9dd3a42f64ed1fcef9696675df3a5c953a: Status 404 returned error can't find the container with id f710dd87a10bcb0045792e084e350e9dd3a42f64ed1fcef9696675df3a5c953a
Oct 30 18:32:28.394618 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.394579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:32:28.448556 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.448521    2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:32:28.453805 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.453772    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:28.464947 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:28.464908    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0072d4_229e_452e_a533_7bf54e6f5ba4.slice/crio-e6c9f64e6f4e6e4ccc1bba333ff0c552523b0b069771fe7fad4592323e91cb89 WatchSource:0}: Error finding container e6c9f64e6f4e6e4ccc1bba333ff0c552523b0b069771fe7fad4592323e91cb89: Status 404 returned error can't find the container with id e6c9f64e6f4e6e4ccc1bba333ff0c552523b0b069771fe7fad4592323e91cb89
Oct 30 18:32:28.530090 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.530030    2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:32:28.591657 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.591551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:28.591831 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.591729    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:28.591831 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.591801    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs podName:48282811-970d-41fb-8745-54023a7558d3 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:29.591785693 +0000 UTC m=+3.073485246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs") pod "network-metrics-daemon-ngsb2" (UID: "48282811-970d-41fb-8745-54023a7558d3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:28.600783 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:28.600733    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701c1b5b_7bf7_4f11_9623_f50f2181c384.slice/crio-4c03044f2cb62c6599adfec04fca36cc84654335b058c6c57c251246a2265f93 WatchSource:0}: Error finding container 4c03044f2cb62c6599adfec04fca36cc84654335b058c6c57c251246a2265f93: Status 404 returned error can't find the container with id 4c03044f2cb62c6599adfec04fca36cc84654335b058c6c57c251246a2265f93
Oct 30 18:32:28.692645 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.692584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btss5\" (UniqueName: \"kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5\") pod \"network-check-target-n52lm\" (UID: \"91317f6c-d730-44f1-a5a1-96ba9c1a6aec\") " pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:28.692731 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.692649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:28.692830 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.692811    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:28.692889 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.692845    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:32:28.692889 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.692875    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:32:28.692984 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.692892    2581 projected.go:196] Error preparing data for projected volume kube-api-access-btss5 for pod openshift-network-diagnostics/network-check-target-n52lm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:28.692984 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.692894    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret podName:faeb3114-7d5c-4d94-a00c-7af6f1c175df nodeName:}" failed. No retries permitted until 2026-10-30 18:32:29.692877819 +0000 UTC m=+3.174577351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret") pod "global-pull-secret-syncer-qmbfz" (UID: "faeb3114-7d5c-4d94-a00c-7af6f1c175df") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:28.692984 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:28.692952    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5 podName:91317f6c-d730-44f1-a5a1-96ba9c1a6aec nodeName:}" failed. No retries permitted until 2026-10-30 18:32:29.692933986 +0000 UTC m=+3.174633538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-btss5" (UniqueName: "kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5") pod "network-check-target-n52lm" (UID: "91317f6c-d730-44f1-a5a1-96ba9c1a6aec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:28.695695 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:32:28.695629    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24897d0a_ab73_4e75_ac67_c5f5103571d0.slice/crio-ef5afd8e9e9720b2ba8ae424dc4281150f04934e0e038d3807667ff3798f93b1 WatchSource:0}: Error finding container ef5afd8e9e9720b2ba8ae424dc4281150f04934e0e038d3807667ff3798f93b1: Status 404 returned error can't find the container with id ef5afd8e9e9720b2ba8ae424dc4281150f04934e0e038d3807667ff3798f93b1
Oct 30 18:32:28.936034 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:28.935898    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:32:29.031103 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.031023    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:27:28 +0000 UTC" deadline="2028-08-02 08:08:45.68619521 +0000 UTC"
Oct 30 18:32:29.031103 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.031092    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15397h36m16.655105861s"
Oct 30 18:32:29.104788 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.104692    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h64hp" event={"ID":"9c151e21-28d5-4d4a-8b95-b92d623582c5","Type":"ContainerStarted","Data":"f710dd87a10bcb0045792e084e350e9dd3a42f64ed1fcef9696675df3a5c953a"}
Oct 30 18:32:29.115189 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.115111    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl" event={"ID":"dd918ffa-ead9-4697-9045-da66e9c7f136","Type":"ContainerStarted","Data":"5b07f85050d7d978a0a56fd257596e801ed11cfce1425a5f08d52e1c44e9b03e"}
Oct 30 18:32:29.127828 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.127776    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal" event={"ID":"b46186b28ca7c09f5723c9f8bbbafe8c","Type":"ContainerStarted","Data":"d2edb2b019e72eea9d832b421121189a8b4a30a8107ddeca4cb7a3a7726b401f"}
Oct 30 18:32:29.136037 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.135972    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z4csl" event={"ID":"24897d0a-ab73-4e75-ac67-c5f5103571d0","Type":"ContainerStarted","Data":"ef5afd8e9e9720b2ba8ae424dc4281150f04934e0e038d3807667ff3798f93b1"}
Oct 30 18:32:29.139471 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.139427    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h7gnp" event={"ID":"701c1b5b-7bf7-4f11-9623-f50f2181c384","Type":"ContainerStarted","Data":"4c03044f2cb62c6599adfec04fca36cc84654335b058c6c57c251246a2265f93"}
Oct 30 18:32:29.149464 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.149311    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f" event={"ID":"3f0072d4-229e-452e-a533-7bf54e6f5ba4","Type":"ContainerStarted","Data":"e6c9f64e6f4e6e4ccc1bba333ff0c552523b0b069771fe7fad4592323e91cb89"}
Oct 30 18:32:29.174907 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.174858    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlnz5" event={"ID":"ba289df1-672b-4bf2-85f6-eaffd2269631","Type":"ContainerStarted","Data":"1836930c4aebb50d6804106a36afab66655a250c49b0a589394af2f4148e7d79"}
Oct 30 18:32:29.195441 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.195201    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5nnbq" event={"ID":"884761af-7ea5-4be1-bd94-e736df037d21","Type":"ContainerStarted","Data":"e4ac273964b9f14facbe2259b965282a07321cf58f2e353bcdad262a4f216902"}
Oct 30 18:32:29.208365 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.208309    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9mqb5" event={"ID":"382d4192-0bfc-4626-820d-5cf183296180","Type":"ContainerStarted","Data":"923d622a4a41829901a91e68bcb94753674b1c2685d212f58f13cb0c53c44828"}
Oct 30 18:32:29.230447 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.230192    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8vlwv" event={"ID":"41efb608-1c74-4ab4-9693-53495bfc26b9","Type":"ContainerStarted","Data":"a9f27f3523180933c4e6aa107ff07ce589d7d707440f9b74848dff1fe3249002"}
Oct 30 18:32:29.239609 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.239555    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal" event={"ID":"c870f7ea63961faf808ec834ef3f426f","Type":"ContainerStarted","Data":"6eccfeda5b686bab80f177d0880cb511c20ed2ceae94660875b10d7c7846a66c"}
Oct 30 18:32:29.608370 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.608144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:29.608590 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:29.608422    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:29.608590 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:29.608512    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs podName:48282811-970d-41fb-8745-54023a7558d3 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:31.608491595 +0000 UTC m=+5.090191135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs") pod "network-metrics-daemon-ngsb2" (UID: "48282811-970d-41fb-8745-54023a7558d3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:29.708993 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.708941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btss5\" (UniqueName: \"kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5\") pod \"network-check-target-n52lm\" (UID: \"91317f6c-d730-44f1-a5a1-96ba9c1a6aec\") " pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:29.709224 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:29.709033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:29.709224 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:29.709164    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:32:29.709224 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:29.709195    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:29.709224 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:29.709200    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:32:29.709224 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:29.709225    2581 projected.go:196] Error preparing data for projected volume kube-api-access-btss5 for pod openshift-network-diagnostics/network-check-target-n52lm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:29.709507 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:29.709284    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret podName:faeb3114-7d5c-4d94-a00c-7af6f1c175df nodeName:}" failed. No retries permitted until 2026-10-30 18:32:31.709254256 +0000 UTC m=+5.190953806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret") pod "global-pull-secret-syncer-qmbfz" (UID: "faeb3114-7d5c-4d94-a00c-7af6f1c175df") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:29.709507 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:29.709307    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5 podName:91317f6c-d730-44f1-a5a1-96ba9c1a6aec nodeName:}" failed. No retries permitted until 2026-10-30 18:32:31.709297193 +0000 UTC m=+5.190996729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-btss5" (UniqueName: "kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5") pod "network-check-target-n52lm" (UID: "91317f6c-d730-44f1-a5a1-96ba9c1a6aec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:30.031959 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:30.031847    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:27:28 +0000 UTC" deadline="2028-05-07 12:02:33.44738648 +0000 UTC"
Oct 30 18:32:30.031959 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:30.031896    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13313h30m3.41549617s"
Oct 30 18:32:30.070225 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:30.066636    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:30.070225 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:30.066845    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n52lm" podUID="91317f6c-d730-44f1-a5a1-96ba9c1a6aec"
Oct 30 18:32:30.070225 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:30.067248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:30.070225 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:30.067276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:30.070225 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:30.067464    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngsb2" podUID="48282811-970d-41fb-8745-54023a7558d3"
Oct 30 18:32:30.070225 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:30.067570    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbfz" podUID="faeb3114-7d5c-4d94-a00c-7af6f1c175df"
Oct 30 18:32:31.090053 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:31.090008    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:32:31.630355 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:31.630299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:31.630743 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:31.630537    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:31.630743 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:31.630629    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs podName:48282811-970d-41fb-8745-54023a7558d3 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:35.630604544 +0000 UTC m=+9.112304083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs") pod "network-metrics-daemon-ngsb2" (UID: "48282811-970d-41fb-8745-54023a7558d3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:31.731575 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:31.731530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btss5\" (UniqueName: \"kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5\") pod \"network-check-target-n52lm\" (UID: \"91317f6c-d730-44f1-a5a1-96ba9c1a6aec\") " pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:31.731791 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:31.731608    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:31.731874 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:31.731789    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:31.731987 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:31.731872    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret podName:faeb3114-7d5c-4d94-a00c-7af6f1c175df nodeName:}" failed. No retries permitted until 2026-10-30 18:32:35.731850862 +0000 UTC m=+9.213550409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret") pod "global-pull-secret-syncer-qmbfz" (UID: "faeb3114-7d5c-4d94-a00c-7af6f1c175df") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:31.732446 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:31.732419    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:32:31.732446 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:31.732448    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:32:31.732626 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:31.732462    2581 projected.go:196] Error preparing data for projected volume kube-api-access-btss5 for pod openshift-network-diagnostics/network-check-target-n52lm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:31.732626 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:31.732517    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5 podName:91317f6c-d730-44f1-a5a1-96ba9c1a6aec nodeName:}" failed. No retries permitted until 2026-10-30 18:32:35.732499376 +0000 UTC m=+9.214198923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-btss5" (UniqueName: "kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5") pod "network-check-target-n52lm" (UID: "91317f6c-d730-44f1-a5a1-96ba9c1a6aec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:32.066715 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:32.066615    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:32.066894 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:32.066792    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n52lm" podUID="91317f6c-d730-44f1-a5a1-96ba9c1a6aec"
Oct 30 18:32:32.067327 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:32.067299    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:32.067457 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:32.067437    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbfz" podUID="faeb3114-7d5c-4d94-a00c-7af6f1c175df"
Oct 30 18:32:32.067532 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:32.067516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:32.067691 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:32.067604    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngsb2" podUID="48282811-970d-41fb-8745-54023a7558d3"
Oct 30 18:32:33.266931 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:33.266839    2581 generic.go:356] "Generic (PLEG): container finished" podID="ba289df1-672b-4bf2-85f6-eaffd2269631" containerID="675214b2cb619c196ce2f6fffb7d3be14bd5111b94cde32418725a077c0aaa13" exitCode=0
Oct 30 18:32:33.267569 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:33.267056    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlnz5" event={"ID":"ba289df1-672b-4bf2-85f6-eaffd2269631","Type":"ContainerDied","Data":"675214b2cb619c196ce2f6fffb7d3be14bd5111b94cde32418725a077c0aaa13"}
Oct 30 18:32:33.272856 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:33.272807    2581 generic.go:356] "Generic (PLEG): container finished" podID="b46186b28ca7c09f5723c9f8bbbafe8c" containerID="2b1b5ccbe0646a1b302960e70268e590b8dccf82676a0a22c00e4e2dc6e042a0" exitCode=0
Oct 30 18:32:33.273033 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:33.272879    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal" event={"ID":"b46186b28ca7c09f5723c9f8bbbafe8c","Type":"ContainerDied","Data":"2b1b5ccbe0646a1b302960e70268e590b8dccf82676a0a22c00e4e2dc6e042a0"}
Oct 30 18:32:34.066457 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:34.066409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:34.066714 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:34.066595    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngsb2" podUID="48282811-970d-41fb-8745-54023a7558d3"
Oct 30 18:32:34.067073 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:34.067052    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:34.067221 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:34.067168    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n52lm" podUID="91317f6c-d730-44f1-a5a1-96ba9c1a6aec"
Oct 30 18:32:34.067221 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:34.067216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:34.067353 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:34.067292    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbfz" podUID="faeb3114-7d5c-4d94-a00c-7af6f1c175df"
Oct 30 18:32:34.295151 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:34.295070    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8vlwv" event={"ID":"41efb608-1c74-4ab4-9693-53495bfc26b9","Type":"ContainerStarted","Data":"e2c7a5c875c712cb0e8b4ef822785a10303036c863a69c203881d162d9d98272"}
Oct 30 18:32:34.303585 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:34.303541    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal" event={"ID":"c870f7ea63961faf808ec834ef3f426f","Type":"ContainerStarted","Data":"d70eb30500b4fcc82d492c55748e0e79be2d05c2d42bb464fc33606160f3dc0f"}
Oct 30 18:32:34.316162 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:34.312777    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal" event={"ID":"b46186b28ca7c09f5723c9f8bbbafe8c","Type":"ContainerStarted","Data":"bfb9b26634939c8f9ea96d61a4ebf78bc17647e7406601bf191ddff81366b37f"}
Oct 30 18:32:34.316162 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:34.315091    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8vlwv" podStartSLOduration=2.509079709 podStartE2EDuration="7.315069594s" totalImagesPullingTime="4.805989885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:34.314019637 +0000 UTC m=+7.795719238" watchObservedRunningTime="2026-10-30 18:32:34.315069594 +0000 UTC m=+7.796769150"
Oct 30 18:32:34.334712 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:34.333445    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal" podStartSLOduration=7.333420066 podStartE2EDuration="7.333420066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:34.33306029 +0000 UTC m=+7.814759844" watchObservedRunningTime="2026-10-30 18:32:34.333420066 +0000 UTC m=+7.815119619"
Oct 30 18:32:34.352762 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:34.351851    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal" podStartSLOduration=7.351826499 podStartE2EDuration="7.351826499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:34.351381939 +0000 UTC m=+7.833081492" watchObservedRunningTime="2026-10-30 18:32:34.351826499 +0000 UTC m=+7.833526071"
Oct 30 18:32:35.327515 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:35.327472    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9mqb5" event={"ID":"382d4192-0bfc-4626-820d-5cf183296180","Type":"ContainerStarted","Data":"500dda24ba48d02a3a28c2c9cb39e345121803fb1838871c20d1c9b984d0ae50"}
Oct 30 18:32:35.675967 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:35.675283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:35.675967 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:35.675478    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:35.675967 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:35.675551    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs podName:48282811-970d-41fb-8745-54023a7558d3 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:43.675531134 +0000 UTC m=+17.157230664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs") pod "network-metrics-daemon-ngsb2" (UID: "48282811-970d-41fb-8745-54023a7558d3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:35.776056 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:35.775952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btss5\" (UniqueName: \"kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5\") pod \"network-check-target-n52lm\" (UID: \"91317f6c-d730-44f1-a5a1-96ba9c1a6aec\") " pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:35.776056 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:35.776038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:35.776292 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:35.776234    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:35.776327 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:35.776313    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret podName:faeb3114-7d5c-4d94-a00c-7af6f1c175df nodeName:}" failed. No retries permitted until 2026-10-30 18:32:43.776289192 +0000 UTC m=+17.257988736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret") pod "global-pull-secret-syncer-qmbfz" (UID: "faeb3114-7d5c-4d94-a00c-7af6f1c175df") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:35.776725 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:35.776709    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:32:35.776725 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:35.776728    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:32:35.776842 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:35.776739    2581 projected.go:196] Error preparing data for projected volume kube-api-access-btss5 for pod openshift-network-diagnostics/network-check-target-n52lm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:35.776842 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:35.776780    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5 podName:91317f6c-d730-44f1-a5a1-96ba9c1a6aec nodeName:}" failed. No retries permitted until 2026-10-30 18:32:43.776768065 +0000 UTC m=+17.258467609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-btss5" (UniqueName: "kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5") pod "network-check-target-n52lm" (UID: "91317f6c-d730-44f1-a5a1-96ba9c1a6aec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:36.066355 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:36.066243    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:36.066625 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:36.066429    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n52lm" podUID="91317f6c-d730-44f1-a5a1-96ba9c1a6aec"
Oct 30 18:32:36.067031 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:36.067003    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:36.067213 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:36.067159    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbfz" podUID="faeb3114-7d5c-4d94-a00c-7af6f1c175df"
Oct 30 18:32:36.067347 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:36.067252    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:36.067423 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:36.067351    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngsb2" podUID="48282811-970d-41fb-8745-54023a7558d3"
Oct 30 18:32:36.335700 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:36.335322    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl" event={"ID":"dd918ffa-ead9-4697-9045-da66e9c7f136","Type":"ContainerStarted","Data":"c1f314bef55d46f0f5c9ebf8321788e00bf7d2c5c84096d27bfc04ee02fdce78"}
Oct 30 18:32:37.041385 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:37.041337    2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:32:37.056414 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:37.056238    2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:32:37.04136371Z","UUID":"8120d261-91b9-4f91-95c6-1f65528545d8","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:32:37.058681 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:37.058644    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:32:37.058681 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:37.058689    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:32:37.246192 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:37.246151    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:32:37.247444 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:37.247415    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:32:37.265884 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:37.265789    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9mqb5" podStartSLOduration=4.326279272 podStartE2EDuration="10.265757743s" totalImagesPullingTime="5.939478471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:35.346954518 +0000 UTC m=+8.828654072" watchObservedRunningTime="2026-10-30 18:32:37.265757743 +0000 UTC m=+10.747457298"
Oct 30 18:32:37.340375 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:37.340104    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl" event={"ID":"dd918ffa-ead9-4697-9045-da66e9c7f136","Type":"ContainerStarted","Data":"4dd73736dfd754e8b166ef440f5e3a2c88d741cae4502a932c7a75713700e159"}
Oct 30 18:32:37.344290 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:37.344206    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5nnbq" event={"ID":"884761af-7ea5-4be1-bd94-e736df037d21","Type":"ContainerStarted","Data":"0bba1a7713ee3b9a5a7ef073720b99de9160b4c412fe90d416779ee802a1d8fe"}
Oct 30 18:32:38.067167 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:38.065984    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:38.067167 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:38.066337    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n52lm" podUID="91317f6c-d730-44f1-a5a1-96ba9c1a6aec"
Oct 30 18:32:38.067167 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:38.066772    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:38.067167 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:38.066892    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbfz" podUID="faeb3114-7d5c-4d94-a00c-7af6f1c175df"
Oct 30 18:32:38.067167 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:38.066964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:38.067167 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:38.067049    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngsb2" podUID="48282811-970d-41fb-8745-54023a7558d3"
Oct 30 18:32:38.350228 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:38.350169    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h64hp" event={"ID":"9c151e21-28d5-4d4a-8b95-b92d623582c5","Type":"ContainerStarted","Data":"683245253c9e1d605690edd4477875ce0a235b48a7f342eb0bccb88fcd829979"}
Oct 30 18:32:38.354490 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:38.353834    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl" event={"ID":"dd918ffa-ead9-4697-9045-da66e9c7f136","Type":"ContainerStarted","Data":"673bce2e29aaabf24f35878063331b9cd7b108c0ff9d99b94ca25871b2c32003"}
Oct 30 18:32:38.356356 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:38.356320    2581 generic.go:356] "Generic (PLEG): container finished" podID="ba289df1-672b-4bf2-85f6-eaffd2269631" containerID="fc6354d7d79dbec86a7f7fdf22a6d86f62484779ebd157fcfaf02546e47c6326" exitCode=0
Oct 30 18:32:38.357230 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:38.357196    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlnz5" event={"ID":"ba289df1-672b-4bf2-85f6-eaffd2269631","Type":"ContainerDied","Data":"fc6354d7d79dbec86a7f7fdf22a6d86f62484779ebd157fcfaf02546e47c6326"}
Oct 30 18:32:38.368501 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:38.367961    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-5nnbq" podStartSLOduration=2.548747268 podStartE2EDuration="11.36793318s" totalImagesPullingTime="8.819185912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:37.366740359 +0000 UTC m=+10.848439913" watchObservedRunningTime="2026-10-30 18:32:38.36793318 +0000 UTC m=+11.849632739"
Oct 30 18:32:38.368501 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:38.368221    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-h64hp" podStartSLOduration=2.408131214 podStartE2EDuration="11.368208618s" totalImagesPullingTime="8.960077404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:38.367326547 +0000 UTC m=+11.849026118" watchObservedRunningTime="2026-10-30 18:32:38.368208618 +0000 UTC m=+11.849908174"
Oct 30 18:32:38.388613 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:38.388534    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl" podStartSLOduration=2.033272152 podStartE2EDuration="11.388515034s" totalImagesPullingTime="9.355242882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:38.387809286 +0000 UTC m=+11.869508864" watchObservedRunningTime="2026-10-30 18:32:38.388515034 +0000 UTC m=+11.870214589"
Oct 30 18:32:39.361007 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:39.360959    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h7gnp" event={"ID":"701c1b5b-7bf7-4f11-9623-f50f2181c384","Type":"ContainerStarted","Data":"9472865c38e639929fe2870ad393b27d543984ec235e678fa7b1ad47d3e8ea4c"}
Oct 30 18:32:39.364012 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:39.363973    2581 generic.go:356] "Generic (PLEG): container finished" podID="ba289df1-672b-4bf2-85f6-eaffd2269631" containerID="490753a085bb889bca947b6d43d3d28e4aa2b016ad78ce0e3ac03082b29fc288" exitCode=0
Oct 30 18:32:39.364220 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:39.364109    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlnz5" event={"ID":"ba289df1-672b-4bf2-85f6-eaffd2269631","Type":"ContainerDied","Data":"490753a085bb889bca947b6d43d3d28e4aa2b016ad78ce0e3ac03082b29fc288"}
Oct 30 18:32:40.065816 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:40.065781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:40.066043 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:40.065856    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:40.066043 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:40.066002    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n52lm" podUID="91317f6c-d730-44f1-a5a1-96ba9c1a6aec"
Oct 30 18:32:40.066174 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:40.066041    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:40.066246 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:40.066214    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngsb2" podUID="48282811-970d-41fb-8745-54023a7558d3"
Oct 30 18:32:40.066341 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:40.066319    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbfz" podUID="faeb3114-7d5c-4d94-a00c-7af6f1c175df"
Oct 30 18:32:40.369269 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:40.369164    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z4csl" event={"ID":"24897d0a-ab73-4e75-ac67-c5f5103571d0","Type":"ContainerStarted","Data":"8d149883459e03627508f74c9ac78ab5085093d1bea2ba1e8971924a27683513"}
Oct 30 18:32:40.389176 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:40.389083    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z4csl" podStartSLOduration=2.031424328 podStartE2EDuration="13.38906625s" totalImagesPullingTime="11.357641922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:40.388480401 +0000 UTC m=+13.870179957" watchObservedRunningTime="2026-10-30 18:32:40.38906625 +0000 UTC m=+13.870765856"
Oct 30 18:32:41.374843 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:41.374803    2581 generic.go:356] "Generic (PLEG): container finished" podID="ba289df1-672b-4bf2-85f6-eaffd2269631" containerID="b69f0ad16a280bce20872a34cdb0d1550beaad7485e5f118a924cae5d41cd7c2" exitCode=0
Oct 30 18:32:41.375465 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:41.374896    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlnz5" event={"ID":"ba289df1-672b-4bf2-85f6-eaffd2269631","Type":"ContainerDied","Data":"b69f0ad16a280bce20872a34cdb0d1550beaad7485e5f118a924cae5d41cd7c2"}
Oct 30 18:32:41.405392 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:41.405312    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h7gnp" podStartSLOduration=5.496328677 podStartE2EDuration="14.405292365s" totalImagesPullingTime="8.908963688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:40.405521807 +0000 UTC m=+13.887221360" watchObservedRunningTime="2026-10-30 18:32:41.405292365 +0000 UTC m=+14.886991917"
Oct 30 18:32:42.066489 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:42.066435    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:42.066758 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:42.066603    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n52lm" podUID="91317f6c-d730-44f1-a5a1-96ba9c1a6aec"
Oct 30 18:32:42.067087 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:42.067058    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:42.067223 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:42.067187    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbfz" podUID="faeb3114-7d5c-4d94-a00c-7af6f1c175df"
Oct 30 18:32:42.067276 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:42.067255    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:42.067368 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:42.067342    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngsb2" podUID="48282811-970d-41fb-8745-54023a7558d3"
Oct 30 18:32:43.747169 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:43.746798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:43.747706 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:43.747015    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:43.747706 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:43.747278    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs podName:48282811-970d-41fb-8745-54023a7558d3 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:59.747251771 +0000 UTC m=+33.228951445 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs") pod "network-metrics-daemon-ngsb2" (UID: "48282811-970d-41fb-8745-54023a7558d3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:32:43.847948 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:43.847901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:43.848248 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:43.847982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btss5\" (UniqueName: \"kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5\") pod \"network-check-target-n52lm\" (UID: \"91317f6c-d730-44f1-a5a1-96ba9c1a6aec\") " pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:43.848248 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:43.848082    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:43.848248 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:43.848110    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:32:43.848248 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:43.848154    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:32:43.848248 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:43.848170    2581 projected.go:196] Error preparing data for projected volume kube-api-access-btss5 for pod openshift-network-diagnostics/network-check-target-n52lm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:43.848248 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:43.848194    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret podName:faeb3114-7d5c-4d94-a00c-7af6f1c175df nodeName:}" failed. No retries permitted until 2026-10-30 18:32:59.84817205 +0000 UTC m=+33.329871594 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret") pod "global-pull-secret-syncer-qmbfz" (UID: "faeb3114-7d5c-4d94-a00c-7af6f1c175df") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:32:43.848248 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:43.848247    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5 podName:91317f6c-d730-44f1-a5a1-96ba9c1a6aec nodeName:}" failed. No retries permitted until 2026-10-30 18:32:59.8482303 +0000 UTC m=+33.329929848 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-btss5" (UniqueName: "kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5") pod "network-check-target-n52lm" (UID: "91317f6c-d730-44f1-a5a1-96ba9c1a6aec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:32:44.066413 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:44.066294    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:44.066413 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:44.066346    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:44.066413 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:44.066346    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:44.066767 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:44.066427    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbfz" podUID="faeb3114-7d5c-4d94-a00c-7af6f1c175df"
Oct 30 18:32:44.066767 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:44.066538    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngsb2" podUID="48282811-970d-41fb-8745-54023a7558d3"
Oct 30 18:32:44.066767 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:44.066643    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n52lm" podUID="91317f6c-d730-44f1-a5a1-96ba9c1a6aec"
Oct 30 18:32:44.383231 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:44.383171    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f" event={"ID":"3f0072d4-229e-452e-a533-7bf54e6f5ba4","Type":"ContainerStarted","Data":"252012a191a0e31ab60e45b8efa9b6a72003ddf04847cc9f2258f4cecabacef4"}
Oct 30 18:32:44.386712 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:44.386474    2581 generic.go:356] "Generic (PLEG): container finished" podID="ba289df1-672b-4bf2-85f6-eaffd2269631" containerID="17a3c45b31b79029eec4125100883de75ad5e63af6508ca0876b7672dfefb1ac" exitCode=0
Oct 30 18:32:44.386712 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:44.386550    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlnz5" event={"ID":"ba289df1-672b-4bf2-85f6-eaffd2269631","Type":"ContainerDied","Data":"17a3c45b31b79029eec4125100883de75ad5e63af6508ca0876b7672dfefb1ac"}
Oct 30 18:32:45.393042 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:45.392770    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f" event={"ID":"3f0072d4-229e-452e-a533-7bf54e6f5ba4","Type":"ContainerStarted","Data":"757397cf8fa713b59c3cfe05ed14045beb85f93591a9c996765494368f0091a8"}
Oct 30 18:32:45.393722 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:45.393116    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f" event={"ID":"3f0072d4-229e-452e-a533-7bf54e6f5ba4","Type":"ContainerStarted","Data":"7845dc5eaa669cd57888f47d290f013164b4e5b0a1518a7878d066eb25e2a1de"}
Oct 30 18:32:45.393722 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:45.393188    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f" event={"ID":"3f0072d4-229e-452e-a533-7bf54e6f5ba4","Type":"ContainerStarted","Data":"a0b08661d62486e6e639a39a012aab9bb1735a66ae608fb10c486f5642d6649e"}
Oct 30 18:32:45.393722 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:45.393234    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f" event={"ID":"3f0072d4-229e-452e-a533-7bf54e6f5ba4","Type":"ContainerStarted","Data":"8fff0a42439c8406de3daf60a1cefa759086f0489af92949ab9789498bcba0f9"}
Oct 30 18:32:45.393722 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:45.393288    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f" event={"ID":"3f0072d4-229e-452e-a533-7bf54e6f5ba4","Type":"ContainerStarted","Data":"ca01bdc107938be3e1be42df1cc0ccf3122ffb59b35eecd20f554b4d4ef6dc1e"}
Oct 30 18:32:45.396148 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:45.396090    2581 generic.go:356] "Generic (PLEG): container finished" podID="ba289df1-672b-4bf2-85f6-eaffd2269631" containerID="0423830d87096871cc2a9a94558ff44dd379243e2527f1506efc8941ae686540" exitCode=0
Oct 30 18:32:45.396308 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:45.396171    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlnz5" event={"ID":"ba289df1-672b-4bf2-85f6-eaffd2269631","Type":"ContainerDied","Data":"0423830d87096871cc2a9a94558ff44dd379243e2527f1506efc8941ae686540"}
Oct 30 18:32:46.066465 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:46.066421    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:46.066695 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:46.066438    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:46.066695 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:46.066544    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbfz" podUID="faeb3114-7d5c-4d94-a00c-7af6f1c175df"
Oct 30 18:32:46.066695 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:46.066568    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:46.066695 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:46.066635    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n52lm" podUID="91317f6c-d730-44f1-a5a1-96ba9c1a6aec"
Oct 30 18:32:46.066839 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:46.066703    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngsb2" podUID="48282811-970d-41fb-8745-54023a7558d3"
Oct 30 18:32:46.402910 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:46.402795    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlnz5" event={"ID":"ba289df1-672b-4bf2-85f6-eaffd2269631","Type":"ContainerStarted","Data":"24605147f1af38cb969ee0d180a842a01b397a1e93aea9bc921266e56ff4f1ab"}
Oct 30 18:32:46.436489 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:46.436427    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vlnz5" podStartSLOduration=7.802130645 podStartE2EDuration="19.436412218s" totalImagesPullingTime="11.634281573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:46.436207578 +0000 UTC m=+19.917907393" watchObservedRunningTime="2026-10-30 18:32:46.436412218 +0000 UTC m=+19.918111770"
Oct 30 18:32:47.411725 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:47.411673    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f" event={"ID":"3f0072d4-229e-452e-a533-7bf54e6f5ba4","Type":"ContainerStarted","Data":"d2a43159bd64e1dec5836f5fcb5f756e15188cb9d277d5cfa927cd1705de6771"}
Oct 30 18:32:48.066772 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:48.066721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:48.067050 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:48.066721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:48.067050 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:48.066876    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngsb2" podUID="48282811-970d-41fb-8745-54023a7558d3"
Oct 30 18:32:48.067050 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:48.066721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:48.067050 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:48.066908    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n52lm" podUID="91317f6c-d730-44f1-a5a1-96ba9c1a6aec"
Oct 30 18:32:48.067050 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:48.067004    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbfz" podUID="faeb3114-7d5c-4d94-a00c-7af6f1c175df"
Oct 30 18:32:48.608686 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:48.608643    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vlwv_41efb608-1c74-4ab4-9693-53495bfc26b9/konnectivity-agent/0.log" limit=1024
Oct 30 18:32:48.778911 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:48.778870    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vlwv_41efb608-1c74-4ab4-9693-53495bfc26b9/konnectivity-agent/0.log" limit=1024
Oct 30 18:32:49.421438 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:49.421179    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f" event={"ID":"3f0072d4-229e-452e-a533-7bf54e6f5ba4","Type":"ContainerStarted","Data":"9f0f8c1e8be83e53b2a9ce40673e4eab72c108095eabf006ee9b029b9e72296d"}
Oct 30 18:32:49.421698 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:49.421498    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:49.421698 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:49.421552    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:49.441781 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:49.441746    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:49.461922 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:49.461862    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f" podStartSLOduration=6.65005065 podStartE2EDuration="22.461843964s" totalImagesPullingTime="15.811793314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:49.460900496 +0000 UTC m=+22.942600049" watchObservedRunningTime="2026-10-30 18:32:49.461843964 +0000 UTC m=+22.943543516"
Oct 30 18:32:50.066599 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:50.066546    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:50.066599 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:50.066591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:50.067348 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:50.066591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:50.067348 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:50.066669    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbfz" podUID="faeb3114-7d5c-4d94-a00c-7af6f1c175df"
Oct 30 18:32:50.067348 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:50.066798    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngsb2" podUID="48282811-970d-41fb-8745-54023a7558d3"
Oct 30 18:32:50.067348 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:50.066892    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n52lm" podUID="91317f6c-d730-44f1-a5a1-96ba9c1a6aec"
Oct 30 18:32:50.424262 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:50.424165    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:50.443414 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:50.443380    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:32:51.012425 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:51.012312    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n52lm"]
Oct 30 18:32:51.012658 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:51.012526    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:51.012716 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:51.012669    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n52lm" podUID="91317f6c-d730-44f1-a5a1-96ba9c1a6aec"
Oct 30 18:32:51.013437 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:51.013401    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ngsb2"]
Oct 30 18:32:51.013605 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:51.013555    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:51.013716 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:51.013686    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngsb2" podUID="48282811-970d-41fb-8745-54023a7558d3"
Oct 30 18:32:51.014273 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:51.014249    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qmbfz"]
Oct 30 18:32:51.014406 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:51.014372    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:51.014510 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:51.014488    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbfz" podUID="faeb3114-7d5c-4d94-a00c-7af6f1c175df"
Oct 30 18:32:53.066715 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:53.066675    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:53.067203 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:53.066675    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:53.067203 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:53.066825    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngsb2" podUID="48282811-970d-41fb-8745-54023a7558d3"
Oct 30 18:32:53.067203 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:53.066676    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:53.067203 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:53.066890    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbfz" podUID="faeb3114-7d5c-4d94-a00c-7af6f1c175df"
Oct 30 18:32:53.067203 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:53.066942    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n52lm" podUID="91317f6c-d730-44f1-a5a1-96ba9c1a6aec"
Oct 30 18:32:55.069923 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:55.069892    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:55.069923 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:55.069908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:55.070475 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:55.070013    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ngsb2" podUID="48282811-970d-41fb-8745-54023a7558d3"
Oct 30 18:32:55.070475 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:55.070061    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:55.070475 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:55.070173    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmbfz" podUID="faeb3114-7d5c-4d94-a00c-7af6f1c175df"
Oct 30 18:32:55.070475 ip-10-0-142-153 kubenswrapper[2581]: E1030 18:32:55.070244    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n52lm" podUID="91317f6c-d730-44f1-a5a1-96ba9c1a6aec"
Oct 30 18:32:56.745717 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:56.745416    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:32:56.746294 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:56.745885    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:32:56.746609 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:56.746589    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:32:56.813077 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:56.813040    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeReady"
Oct 30 18:32:56.813333 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:56.813250    2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 18:32:57.067474 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:57.067428    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:57.067696 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:57.067513    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:57.067696 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:57.067547    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:57.079827 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:57.079789    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:32:57.080332 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:57.079875    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fgz4w\""
Oct 30 18:32:57.080332 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:57.080207    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:32:57.080332 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:57.080274    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9qttr\""
Oct 30 18:32:57.080487 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:57.080357    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z87v4\""
Oct 30 18:32:57.136824 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:57.136776    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:32:57.136824 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:57.136776    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:32:58.420291 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:58.420249    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h64hp_9c151e21-28d5-4d4a-8b95-b92d623582c5/dns-node-resolver/0.log"
Oct 30 18:32:59.221564 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:59.221531    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9mqb5_382d4192-0bfc-4626-820d-5cf183296180/node-ca/0.log"
Oct 30 18:32:59.784014 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:59.783970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:59.789161 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:59.789106    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:59.797192 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:59.797150    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:32:59.884892 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:59.884848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:32:59.885155 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:59.884921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btss5\" (UniqueName: \"kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5\") pod \"network-check-target-n52lm\" (UID: \"91317f6c-d730-44f1-a5a1-96ba9c1a6aec\") " pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:59.888059 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:59.888026    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btss5\" (UniqueName: \"kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5\") pod \"network-check-target-n52lm\" (UID: \"91317f6c-d730-44f1-a5a1-96ba9c1a6aec\") " pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:32:59.888252 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:32:59.888020    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:33:00.087356 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:33:00.087245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:33:00.099661 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:33:00.099625    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:33:00.185474 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:33:00.184501    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ngsb2"]
Oct 30 18:33:00.384548 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:33:00.384285    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qmbfz"]
Oct 30 18:33:00.387210 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:33:00.387176    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaeb3114_7d5c_4d94_a00c_7af6f1c175df.slice/crio-48ce8e3ccfa7d0d988f15696e21a0c8fe0cbcdb83b7d9391b4a9f7fef07197c1 WatchSource:0}: Error finding container 48ce8e3ccfa7d0d988f15696e21a0c8fe0cbcdb83b7d9391b4a9f7fef07197c1: Status 404 returned error can't find the container with id 48ce8e3ccfa7d0d988f15696e21a0c8fe0cbcdb83b7d9391b4a9f7fef07197c1
Oct 30 18:33:00.415946 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:33:00.415906    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n52lm"]
Oct 30 18:33:00.419047 ip-10-0-142-153 kubenswrapper[2581]: W1030 18:33:00.419018    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91317f6c_d730_44f1_a5a1_96ba9c1a6aec.slice/crio-bac59cdfb8b293760afd89f779c4e014c287848c1c496c3389615c50cb81fb61 WatchSource:0}: Error finding container bac59cdfb8b293760afd89f779c4e014c287848c1c496c3389615c50cb81fb61: Status 404 returned error can't find the container with id bac59cdfb8b293760afd89f779c4e014c287848c1c496c3389615c50cb81fb61
Oct 30 18:33:00.448582 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:33:00.448532    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n52lm" event={"ID":"91317f6c-d730-44f1-a5a1-96ba9c1a6aec","Type":"ContainerStarted","Data":"bac59cdfb8b293760afd89f779c4e014c287848c1c496c3389615c50cb81fb61"}
Oct 30 18:33:00.449818 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:33:00.449787    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qmbfz" event={"ID":"faeb3114-7d5c-4d94-a00c-7af6f1c175df","Type":"ContainerStarted","Data":"48ce8e3ccfa7d0d988f15696e21a0c8fe0cbcdb83b7d9391b4a9f7fef07197c1"}
Oct 30 18:33:00.451035 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:33:00.451007    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ngsb2" event={"ID":"48282811-970d-41fb-8745-54023a7558d3","Type":"ContainerStarted","Data":"9ead6483da1d900824c9541d714869bbc0a4b0b6b259dcd1dcf3f4ed05163d00"}
Oct 30 18:33:02.462669 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:33:02.461723    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ngsb2" event={"ID":"48282811-970d-41fb-8745-54023a7558d3","Type":"ContainerStarted","Data":"58935650ab3c327532e7e8569cd56cbbd2921fc76dc4bcb2db69e364fd6e4107"}
Oct 30 18:33:02.462669 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:33:02.461840    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ngsb2" event={"ID":"48282811-970d-41fb-8745-54023a7558d3","Type":"ContainerStarted","Data":"3fd769251b86efd22a002d6a716eb8e3bfe7785124d9c133353c67535b2d707f"}
Oct 30 18:33:02.485175 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:33:02.485069    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ngsb2" podStartSLOduration=33.818382351 podStartE2EDuration="35.485044616s" totalImagesPullingTime="1.666662265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:02.484209504 +0000 UTC m=+35.965909056" watchObservedRunningTime="2026-10-30 18:33:02.485044616 +0000 UTC m=+35.966744170"
Oct 30 18:33:04.469789 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:33:04.469746    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n52lm" event={"ID":"91317f6c-d730-44f1-a5a1-96ba9c1a6aec","Type":"ContainerStarted","Data":"cbd5bed1135ec41edc735b7222f0d6fd4e760b7dfc8f97068b03c89ac3d0ea9c"}
Oct 30 18:33:04.470332 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:33:04.469916    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:33:04.494161 ip-10-0-142-153 kubenswrapper[2581]: I1030 18:33:04.493986    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n52lm" podStartSLOduration=33.674842789 podStartE2EDuration="37.493961933s" totalImagesPullingTime="3.819119144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:04.490836089 +0000 UTC m=+37.972535643" watchObservedRunningTime="2026-10-30 18:33:04.493961933 +0000 UTC m=+37.975661489"
Oct 30 18:33:06.035693 ip-10-0-142-153 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 18:33:06.047606 ip-10-0-142-153 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 18:33:06.047828 ip-10-0-142-153 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 18:33:06.047890 ip-10-0-142-153 systemd[1]: kubelet.service: Consumed 2.779s CPU time.
Oct 30 18:33:06.068002 ip-10-0-142-153 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:33:06.167010 ip-10-0-142-153 kubenswrapper[4372]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:33:06.167010 ip-10-0-142-153 kubenswrapper[4372]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:33:06.167010 ip-10-0-142-153 kubenswrapper[4372]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:33:06.167010 ip-10-0-142-153 kubenswrapper[4372]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:33:06.169897 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169766    4372 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:33:06.169897 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169897    4372 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:33:06.169897 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169901    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:33:06.169897 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169905    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:33:06.169897 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169909    4372 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169912    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169915    4372 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169918    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169921    4372 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169924    4372 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169926    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169929    4372 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169932    4372 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169936    4372 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169939    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169941    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169944    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169947    4372 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169949    4372 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169951    4372 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169954    4372 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169957    4372 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169959    4372 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:33:06.170100 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169962    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169964    4372 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169967    4372 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169969    4372 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169972    4372 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169974    4372 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169977    4372 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169980    4372 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169983    4372 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169989    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169992    4372 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169995    4372 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.169998    4372 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170001    4372 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170003    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170006    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170011    4372 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170016    4372 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170020    4372 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:33:06.170618 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170023    4372 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170025    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170028    4372 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170031    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170034    4372 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170037    4372 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170039    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170042    4372 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170045    4372 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170048    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170050    4372 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170054    4372 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170058    4372 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170062    4372 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170065    4372 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170068    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170070    4372 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170073    4372 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170076    4372 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:33:06.171208 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170079    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170083    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170085    4372 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170088    4372 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170091    4372 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170095    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170098    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170102    4372 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170105    4372 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170108    4372 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170110    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170113    4372 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170116    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170119    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170139    4372 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170143    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170147    4372 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170151    4372 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170155    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:33:06.171699 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170159    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170165    4372 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170170    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170174    4372 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170178    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170182    4372 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170185    4372 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170188    4372 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170191    4372 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170194    4372 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170196    4372 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170199    4372 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170202    4372 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170205    4372 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170207    4372 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170210    4372 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170213    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170216    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170219    4372 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170222    4372 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:33:06.172205 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170225    4372 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170227    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170230    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170234    4372 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170236    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.170239    4372 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171229    4372 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171242    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171246    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171249    4372 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171252    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171255    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171258    4372 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171261    4372 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171265    4372 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171269    4372 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171273    4372 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171276    4372 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171279    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171281    4372 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:33:06.172730 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171285    4372 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171287    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171290    4372 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171293    4372 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171297    4372 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171300    4372 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171303    4372 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171305    4372 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171309    4372 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171312    4372 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171316    4372 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171321    4372 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171325    4372 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171328    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171330    4372 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171333    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171336    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171339    4372 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171342    4372 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:33:06.173264 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171345    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171348    4372 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171351    4372 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171353    4372 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171356    4372 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171358    4372 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171361    4372 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171363    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171368    4372 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171371    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171374    4372 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171377    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171380    4372 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171383    4372 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171386    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171389    4372 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171391    4372 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171394    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171397    4372 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:33:06.173747 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171399    4372 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171402    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171404    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171407    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171409    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171412    4372 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171415    4372 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171417    4372 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171420    4372 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171423    4372 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171425    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171429    4372 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171431    4372 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171434    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171437    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171440    4372 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171443    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171445    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171448    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:33:06.174314 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171451    4372 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171455    4372 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171460    4372 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171464    4372 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171467    4372 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171470    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171473    4372 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171476    4372 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171479    4372 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171481    4372 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171484    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171486    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171489    4372 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171492    4372 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171495    4372 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171497    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171500    4372 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171502    4372 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171505    4372 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:33:06.174832 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171508    4372 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171511    4372 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171514    4372 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171516    4372 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171519    4372 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171522    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171524    4372 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171528    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171531    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171534    4372 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171537    4372 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171540    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171542    4372 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171545    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171547    4372 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.171550    4372 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172181    4372 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172192    4372 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172196    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172200    4372 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:33:06.175396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172205    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172208    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172211    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172214    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172217    4372 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172221    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172224    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172228    4372 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172231    4372 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172234    4372 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172236    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172240    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172243    4372 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172246    4372 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172250    4372 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172253    4372 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172255    4372 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172258    4372 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172260    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172263    4372 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:33:06.175898 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172265    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172269    4372 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172272    4372 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172274    4372 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172278    4372 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172280    4372 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172283    4372 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172285    4372 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172288    4372 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172293    4372 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172295    4372 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172299    4372 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172301    4372 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172304    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172307    4372 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172310    4372 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172314    4372 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172318    4372 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172320    4372 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:33:06.176436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172323    4372 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172326    4372 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172329    4372 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172331    4372 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172334    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172337    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172339    4372 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172342    4372 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172345    4372 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172348    4372 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172350    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172353    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172355    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172358    4372 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172361    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172364    4372 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172367    4372 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172370    4372 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172373    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:33:06.176914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172376    4372 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172378    4372 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172381    4372 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172384    4372 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172388    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172390    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172393    4372 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172395    4372 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172398    4372 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172400    4372 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172403    4372 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172405    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172408    4372 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172411    4372 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172414    4372 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172416    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172419    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172422    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172424    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172428    4372 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:33:06.177421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172431    4372 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172433    4372 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172436    4372 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172438    4372 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172445    4372 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172448    4372 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172451    4372 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172454    4372 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172457    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172460    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172463    4372 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172465    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172468    4372 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172470    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172473    4372 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172478    4372 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172482    4372 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172486    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172488    4372 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:33:06.177927 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172491    4372 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172494    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172496    4372 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172499    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.172501    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172622    4372 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172633    4372 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172641    4372 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172647    4372 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172652    4372 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172656    4372 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172663    4372 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172668    4372 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172672    4372 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172676    4372 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172679    4372 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172683    4372 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172688    4372 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172692    4372 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172695    4372 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172698    4372 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172701    4372 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172704    4372 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:33:06.178427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172710    4372 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172713    4372 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172716    4372 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172719    4372 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172723    4372 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172728    4372 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172732    4372 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172735    4372 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172741    4372 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172745    4372 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172748    4372 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172751    4372 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172755    4372 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172758    4372 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172763    4372 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172766    4372 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172770    4372 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172773    4372 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172777    4372 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172782    4372 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172788    4372 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172791    4372 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172795    4372 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172798    4372 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:33:06.179028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172801    4372 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172807    4372 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172813    4372 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172817    4372 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172821    4372 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172824    4372 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172827    4372 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172830    4372 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172833    4372 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172836    4372 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172839    4372 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172842    4372 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172847    4372 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172851    4372 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172854    4372 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172857    4372 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172861    4372 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172872    4372 flags.go:64] FLAG: --help="false"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172877    4372 flags.go:64] FLAG: --hostname-override="ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172880    4372 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172884    4372 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172887    4372 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172891    4372 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:33:06.179694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172895    4372 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172898    4372 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172901    4372 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172904    4372 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172909    4372 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172913    4372 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172916    4372 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172919    4372 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172922    4372 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172925    4372 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172928    4372 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172932    4372 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172936    4372 flags.go:64] FLAG: --lock-file=""
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172938    4372 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172942    4372 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172945    4372 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172952    4372 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172955    4372 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172958    4372 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172961    4372 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172964    4372 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172968    4372 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172971    4372 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172974    4372 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172981    4372 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:33:06.180378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172985    4372 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172989    4372 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.172998    4372 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173002    4372 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173005    4372 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173009    4372 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173012    4372 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173015    4372 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173018    4372 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173030    4372 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173034    4372 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173039    4372 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173043    4372 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173046    4372 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173049    4372 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173053    4372 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173057    4372 flags.go:64] FLAG: --port="10250"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173060    4372 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173065    4372 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-01ff1c475c929cdac"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173069    4372 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173072    4372 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173075    4372 flags.go:64] FLAG: --register-node="true"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173078    4372 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173082    4372 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173085    4372 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173088    4372 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:33:06.180990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173091    4372 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173096    4372 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173099    4372 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173103    4372 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173106    4372 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173109    4372 flags.go:64] FLAG: --runonce="false"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173113    4372 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173116    4372 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173119    4372 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173158    4372 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173164    4372 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173169    4372 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173173    4372 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173177    4372 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173180    4372 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173184    4372 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173187    4372 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173193    4372 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173196    4372 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173199    4372 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173203    4372 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173210    4372 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173213    4372 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173217    4372 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173228    4372 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:33:06.181696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173232    4372 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173235    4372 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173238    4372 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173241    4372 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173244    4372 flags.go:64] FLAG: --v="2"
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173251    4372 flags.go:64] FLAG: --version="false"
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173256    4372 flags.go:64] FLAG: --vmodule=""
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173265    4372 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173270    4372 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173435    4372 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173440    4372 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173444    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173447    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173449    4372 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173453    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173455    4372 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173458    4372 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173461    4372 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173464    4372 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173467    4372 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173469    4372 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173472    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:33:06.182331 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173475    4372 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173477    4372 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173480    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173483    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173486    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173488    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173493    4372 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173498    4372 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173501    4372 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173507    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173510    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173513    4372 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173516    4372 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173519    4372 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173522    4372 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173524    4372 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173527    4372 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173529    4372 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173532    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:33:06.182902 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173535    4372 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173537    4372 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173540    4372 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173543    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173545    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173548    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173550    4372 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173553    4372 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173555    4372 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173559    4372 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173562    4372 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173564    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173566    4372 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173569    4372 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173572    4372 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173574    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173577    4372 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173580    4372 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173583    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173585    4372 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:33:06.183444 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173588    4372 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173591    4372 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173596    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173599    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173602    4372 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173604    4372 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173607    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173610    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173612    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173615    4372 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173617    4372 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173620    4372 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173623    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173625    4372 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173628    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173631    4372 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173634    4372 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173636    4372 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173639    4372 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:33:06.183931 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173641    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173644    4372 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173646    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173656    4372 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173659    4372 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173661    4372 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173664    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173667    4372 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173669    4372 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173672    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173675    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173677    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173680    4372 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173682    4372 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173685    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173690    4372 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173693    4372 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173696    4372 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173698    4372 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173701    4372 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:33:06.184660 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173704    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:33:06.185480 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173707    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:33:06.185480 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173710    4372 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:33:06.185480 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173713    4372 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:33:06.185480 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173715    4372 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:33:06.185480 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173718    4372 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:33:06.185480 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173721    4372 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:33:06.185480 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173725    4372 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:33:06.185480 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173728    4372 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:33:06.185480 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173731    4372 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:33:06.185480 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173734    4372 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:33:06.185480 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173736    4372 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:33:06.185480 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173739    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:33:06.185480 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173742    4372 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:33:06.185480 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173744    4372 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:33:06.185480 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.173778    4372 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173820    4372 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173827    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173830    4372 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173834    4372 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173837    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173840    4372 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173842    4372 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173845    4372 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173848    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173851    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173853    4372 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173856    4372 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173861    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173863    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173866    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173869    4372 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173871    4372 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173874    4372 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:33:06.185939 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173877    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173879    4372 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173882    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173884    4372 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173887    4372 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173890    4372 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173892    4372 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173895    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173897    4372 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173900    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173902    4372 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173905    4372 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173908    4372 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173910    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173913    4372 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173917    4372 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173920    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173925    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173927    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173930    4372 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:33:06.186623 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173933    4372 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173935    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173938    4372 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173941    4372 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173943    4372 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173946    4372 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173951    4372 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173953    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173956    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173959    4372 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173962    4372 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173964    4372 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173967    4372 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173971    4372 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173974    4372 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173978    4372 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173981    4372 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173984    4372 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173986    4372 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:33:06.187156 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173989    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173992    4372 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173994    4372 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.173997    4372 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174000    4372 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174002    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174005    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174008    4372 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174010    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174014    4372 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174018    4372 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174021    4372 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174024    4372 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174026    4372 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174029    4372 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174031    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174034    4372 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174036    4372 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174038    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174042    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:33:06.187713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174045    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174048    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174051    4372 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174056    4372 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174059    4372 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174061    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174064    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174067    4372 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174070    4372 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174073    4372 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174076    4372 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174078    4372 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174082    4372 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174085    4372 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174087    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174090    4372 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174092    4372 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174095    4372 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174097    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:33:06.188361 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174100    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174102    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174105    4372 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174109    4372 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174113    4372 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174116    4372 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174118    4372 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174144    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174148    4372 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.174152    4372 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.179205    4372 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.179242    4372 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.179265    4372 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.179271    4372 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179342    4372 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179349    4372 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179352    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179356    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179358    4372 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179361    4372 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:33:06.188855 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179364    4372 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179367    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179369    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179372    4372 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179374    4372 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179377    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179379    4372 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179382    4372 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179384    4372 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179387    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179390    4372 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179392    4372 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179395    4372 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179400    4372 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179405    4372 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179409    4372 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179412    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179415    4372 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179418    4372 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:33:06.189426 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179422    4372 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179426    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179430    4372 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179434    4372 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179450    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179453    4372 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179456    4372 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179459    4372 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179462    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179465    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179467    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179470    4372 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179472    4372 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179475    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179478    4372 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179481    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179484    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179487    4372 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179489    4372 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:33:06.190040 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179492    4372 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179494    4372 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179497    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179499    4372 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179502    4372 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179505    4372 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179508    4372 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179511    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179513    4372 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179516    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179519    4372 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179521    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179524    4372 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179527    4372 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179530    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179532    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179535    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179538    4372 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179547    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:33:06.190681 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179550    4372 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179553    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179556    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179558    4372 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179561    4372 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179563    4372 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179566    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179568    4372 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179571    4372 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179573    4372 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179576    4372 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179578    4372 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179581    4372 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179583    4372 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179586    4372 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179589    4372 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179592    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179595    4372 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179598    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179601    4372 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:33:06.191344 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179603    4372 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179606    4372 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179608    4372 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179611    4372 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179614    4372 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179616    4372 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179619    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179621    4372 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179624    4372 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179626    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179629    4372 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179632    4372 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179641    4372 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179643    4372 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179646    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179649    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179652    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179654    4372 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179657    4372 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179659    4372 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:33:06.191851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179662    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179664    4372 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179667    4372 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.179703    4372 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179916    4372 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179922    4372 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179926    4372 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179929    4372 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179932    4372 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179935    4372 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179938    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179941    4372 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179943    4372 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179946    4372 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179948    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179951    4372 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:33:06.192368 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179954    4372 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179957    4372 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179959    4372 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179962    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179964    4372 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179967    4372 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179970    4372 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179972    4372 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179975    4372 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179984    4372 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179987    4372 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179990    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179993    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179996    4372 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.179998    4372 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180001    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180004    4372 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180006    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180009    4372 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:33:06.192779 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180012    4372 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180015    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180018    4372 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180021    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180024    4372 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180027    4372 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180029    4372 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180032    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180034    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180037    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180040    4372 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180042    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180045    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180047    4372 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180050    4372 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180053    4372 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180055    4372 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180058    4372 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180060    4372 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180063    4372 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:33:06.193286 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180066    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180068    4372 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180071    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180080    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180083    4372 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180086    4372 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180088    4372 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180090    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180093    4372 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180095    4372 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180099    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180102    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180105    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180108    4372 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180110    4372 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180113    4372 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180116    4372 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180118    4372 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180136    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180140    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:33:06.194034 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180145    4372 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180149    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180152    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180155    4372 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180158    4372 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180169    4372 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180172    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180175    4372 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180177    4372 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180181    4372 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180184    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180186    4372 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180189    4372 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180192    4372 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180195    4372 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180198    4372 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180207    4372 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180215    4372 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180218    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:33:06.194950 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180220    4372 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180223    4372 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180226    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180228    4372 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180231    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180234    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180237    4372 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180240    4372 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180242    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180245    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180248    4372 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180251    4372 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180253    4372 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180257    4372 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180261    4372 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180264    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:33:06.195443 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.180293    4372 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180336    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180342    4372 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180346    4372 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180349    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180351    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180354    4372 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180357    4372 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180359    4372 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180363    4372 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180366    4372 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180369    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180372    4372 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180374    4372 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180383    4372 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180388    4372 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180391    4372 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180394    4372 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180396    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:33:06.195847 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180399    4372 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180402    4372 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180407    4372 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180409    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180412    4372 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180414    4372 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180417    4372 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180419    4372 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180422    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180425    4372 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180427    4372 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180430    4372 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180433    4372 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180435    4372 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180437    4372 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180440    4372 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180443    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180445    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180448    4372 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:33:06.196384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180451    4372 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180453    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180456    4372 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180459    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180461    4372 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180465    4372 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180470    4372 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180474    4372 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180478    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180491    4372 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180496    4372 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180500    4372 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180504    4372 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180507    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180511    4372 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180517    4372 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180522    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180526    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180531    4372 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180535    4372 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:33:06.196861 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180539    4372 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180543    4372 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180547    4372 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180551    4372 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180555    4372 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180560    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180564    4372 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180569    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180573    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180577    4372 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180581    4372 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180584    4372 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180587    4372 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180589    4372 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180592    4372 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180596    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180598    4372 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180601    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180603    4372 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180606    4372 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:33:06.197384 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180608    4372 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180613    4372 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180625    4372 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180628    4372 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180631    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180635    4372 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180638    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180643    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180645    4372 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180648    4372 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180651    4372 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180654    4372 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180656    4372 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180659    4372 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180661    4372 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180664    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180666    4372 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180669    4372 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180671    4372 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:33:06.197885 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180674    4372 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180677    4372 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180679    4372 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180681    4372 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180684    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180687    4372 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180690    4372 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180693    4372 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180695    4372 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:06.180698    4372 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.180788    4372 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.182155    4372 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.182257    4372 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.182507    4372 server.go:1014] "Starting client certificate rotation"
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.182675    4372 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.182755    4372 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:27:26 +0000 UTC" deadline="2028-04-04 04:40:52.339545614 +0000 UTC"
Oct 30 18:33:06.198380 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.182791    4372 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12514h7m46.156756807s"
Oct 30 18:33:06.198792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.183178    4372 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:33:06.198792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.186174    4372 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:33:06.198792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.187974    4372 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:33:06.198792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.189716    4372 log.go:25] "Validated CRI v1 image API"
Oct 30 18:33:06.198792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.190613    4372 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:33:06.198792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.192499    4372 fs.go:135] Filesystem UUIDs: map[3ba165b1-f104-4863-a297-4f1404590f8e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e26d2272-0779-43f8-b926-34abf9f0af0c:/dev/nvme0n1p4]
Oct 30 18:33:06.198991 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.192527    4372 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1836930c4aebb50d6804106a36afab66655a250c49b0a589394af2f4148e7d79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1836930c4aebb50d6804106a36afab66655a250c49b0a589394af2f4148e7d79/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48ce8e3ccfa7d0d988f15696e21a0c8fe0cbcdb83b7d9391b4a9f7fef07197c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48ce8e3ccfa7d0d988f15696e21a0c8fe0cbcdb83b7d9391b4a9f7fef07197c1/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c03044f2cb62c6599adfec04fca36cc84654335b058c6c57c251246a2265f93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c03044f2cb62c6599adfec04fca36cc84654335b058c6c57c251246a2265f93/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b07f85050d7d978a0a56fd257596e801ed11cfce1425a5f08d52e1c44e9b03e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b07f85050d7d978a0a56fd257596e801ed11cfce1425a5f08d52e1c44e9b03e/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6eccfeda5b686bab80f177d0880cb511c20ed2ceae94660875b10d7c7846a66c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6eccfeda5b686bab80f177d0880cb511c20ed2ceae94660875b10d7c7846a66c/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/923d622a4a41829901a91e68bcb94753674b1c2685d212f58f13cb0c53c44828/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/923d622a4a41829901a91e68bcb94753674b1c2685d212f58f13cb0c53c44828/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ead6483da1d900824c9541d714869bbc0a4b0b6b259dcd1dcf3f4ed05163d00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ead6483da1d900824c9541d714869bbc0a4b0b6b259dcd1dcf3f4ed05163d00/userdata/shm major:0 minor:197 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9f27f3523180933c4e6aa107ff07ce589d7d707440f9b74848dff1fe3249002/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9f27f3523180933c4e6aa107ff07ce589d7d707440f9b74848dff1fe3249002/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bac59cdfb8b293760afd89f779c4e014c287848c1c496c3389615c50cb81fb61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bac59cdfb8b293760afd89f779c4e014c287848c1c496c3389615c50cb81fb61/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2edb2b019e72eea9d832b421121189a8b4a30a8107ddeca4cb7a3a7726b401f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2edb2b019e72eea9d832b421121189a8b4a30a8107ddeca4cb7a3a7726b401f/userdata/shm major:0 minor:59 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4ac273964b9f14facbe2259b965282a07321cf58f2e353bcdad262a4f216902/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4ac273964b9f14facbe2259b965282a07321cf58f2e353bcdad262a4f216902/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6c9f64e6f4e6e4ccc1bba333ff0c552523b0b069771fe7fad4592323e91cb89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6c9f64e6f4e6e4ccc1bba333ff0c552523b0b069771fe7fad4592323e91cb89/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f710dd87a10bcb0045792e084e350e9dd3a42f64ed1fcef9696675df3a5c953a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f710dd87a10bcb0045792e084e350e9dd3a42f64ed1fcef9696675df3a5c953a/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24897d0a-ab73-4e75-ac67-c5f5103571d0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/24897d0a-ab73-4e75-ac67-c5f5103571d0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24897d0a-ab73-4e75-ac67-c5f5103571d0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/24897d0a-ab73-4e75-ac67-c5f5103571d0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24897d0a-ab73-4e75-ac67-c5f5103571d0/volumes/kubernetes.io~projected/kube-api-access-4jh8p:{mountpoint:/var/lib/kubelet/pods/24897d0a-ab73-4e75-ac67-c5f5103571d0/volumes/kubernetes.io~projected/kube-api-access-4jh8p major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/382d4192-0bfc-4626-820d-5cf183296180/volumes/kubernetes.io~projected/kube-api-access-tp7fk:{mountpoint:/var/lib/kubelet/pods/382d4192-0bfc-4626-820d-5cf183296180/volumes/kubernetes.io~projected/kube-api-access-tp7fk major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f0072d4-229e-452e-a533-7bf54e6f5ba4/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/3f0072d4-229e-452e-a533-7bf54e6f5ba4/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f0072d4-229e-452e-a533-7bf54e6f5ba4/volumes/kubernetes.io~projected/kube-api-access-w9sxz:{mountpoint:/var/lib/kubelet/pods/3f0072d4-229e-452e-a533-7bf54e6f5ba4/volumes/kubernetes.io~projected/kube-api-access-w9sxz major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f0072d4-229e-452e-a533-7bf54e6f5ba4/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/3f0072d4-229e-452e-a533-7bf54e6f5ba4/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41efb608-1c74-4ab4-9693-53495bfc26b9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/41efb608-1c74-4ab4-9693-53495bfc26b9/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48282811-970d-41fb-8745-54023a7558d3/volumes/kubernetes.io~projected/kube-api-access-xt8v8:{mountpoint:/var/lib/kubelet/pods/48282811-970d-41fb-8745-54023a7558d3/volumes/kubernetes.io~projected/kube-api-access-xt8v8 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48282811-970d-41fb-8745-54023a7558d3/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/48282811-970d-41fb-8745-54023a7558d3/volumes/kubernetes.io~secret/metrics-certs major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/701c1b5b-7bf7-4f11-9623-f50f2181c384/volumes/kubernetes.io~projected/kube-api-access-pjz6c:{mountpoint:/var/lib/kubelet/pods/701c1b5b-7bf7-4f11-9623-f50f2181c384/volumes/kubernetes.io~projected/kube-api-access-pjz6c major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/884761af-7ea5-4be1-bd94-e736df037d21/volumes/kubernetes.io~projected/kube-api-access-vq592:{mountpoint:/var/lib/kubelet/pods/884761af-7ea5-4be1-bd94-e736df037d21/volumes/kubernetes.io~projected/kube-api-access-vq592 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91317f6c-d730-44f1-a5a1-96ba9c1a6aec/volumes/kubernetes.io~projected/kube-api-access-btss5:{mountpoint:/var/lib/kubelet/pods/91317f6c-d730-44f1-a5a1-96ba9c1a6aec/volumes/kubernetes.io~projected/kube-api-access-btss5 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c151e21-28d5-4d4a-8b95-b92d623582c5/volumes/kubernetes.io~projected/kube-api-access-5ljt9:{mountpoint:/var/lib/kubelet/pods/9c151e21-28d5-4d4a-8b95-b92d623582c5/volumes/kubernetes.io~projected/kube-api-access-5ljt9 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba289df1-672b-4bf2-85f6-eaffd2269631/volumes/kubernetes.io~projected/kube-api-access-wxhdz:{mountpoint:/var/lib/kubelet/pods/ba289df1-672b-4bf2-85f6-eaffd2269631/volumes/kubernetes.io~projected/kube-api-access-wxhdz major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd918ffa-ead9-4697-9045-da66e9c7f136/volumes/kubernetes.io~projected/kube-api-access-mwpb5:{mountpoint:/var/lib/kubelet/pods/dd918ffa-ead9-4697-9045-da66e9c7f136/volumes/kubernetes.io~projected/kube-api-access-mwpb5 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/faeb3114-7d5c-4d94-a00c-7af6f1c175df/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/faeb3114-7d5c-4d94-a00c-7af6f1c175df/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:203 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/fd0221ec771ddb88f4d06dd279ed18f8927a7051a0801968a38b2edd213bb5fd/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/233ccabfd20872d4d004a6bc76282570bdff4d8f6f9c9f9a3f1b69382997f21d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f86c9caad939312ddad4b3c85c183719c08759a193730d7888042742494dd936/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/f4a8f134eaae7e2d6237f20126be2959fa6008d5b489b9412b0a68b4b2b1f34b/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/470f99fdc31f5dd999bd7b405fe4e4993063cb465cdce88a3d6ece66f1f4066a/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f09ae85405cbac59c2b2e99ac7f0afe74ed1105adba18404f14e12d46f69eb58/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/dbb40d2df071565a3d5be06dbdafa7bd062bc665792e2ecf5ffdd39667cc2741/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2e953a08805864015659e30623c99220823b290373c0586645352fd00e222fcb/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e4c6763f81662cdefb671c8d2b237e68ee40cc0ef6294086ddd9c7595a0164fe/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/2726da7d872a6c7480a7a78d240b227a7a194be13538097e31ff30aafaa3ee57/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/054cdd3340f931100ac50ea9117643e93dd3c432887eb3bed907644316d92914/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/17eb692edb74190df6b9077aabc0ce6ecd14e89dd42f2f63ca366857a7fceb17/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9e638e8d04718658cdae1ed5cb28427981f7a46fa1cef1b43238db794a8fccb9/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c0d4564f344dfefa05672ceeb6255fac2e7ff9081abfb9e44eaa732c4cf1f146/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e611f95dcc770f15d672dc617f1516f33c9863f066f71f84f96d283aa8027e94/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-163:{mountpoint:/var/lib/containers/storage/overlay/c8e651ade8534e51201114e9628d0033371a95cc078dfd980efad500194fa523/merged major:0 minor:163 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/c4808f4ff113973623c6172742e412413dc45cd9b593b0d720c92fdac4f72d0e/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f4914b7af225b84408bc2215ae742c649f3262d9f425e9654c8ae0f5c5b9c91d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e18c474eeac2e68c8a3c60db180091393f0d72997043db9cd474834c6ee0be64/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/233972e38c4fb94e570ef91cdea6d8e322b066d2676bbaecd2d485b95e34dddd/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/89c8fc99cb13def5e44269f31f4e149a81eddb5fe145932c0adf0823483f0fb8/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/68fab3228bcf67f78f4569fa10294d2ca06624eb061d8a314e2d9b90096902d6/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/189ad56dc286dc11a7c7e07796cdfe402c0145fd480ac215a46384054e48a586/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/4eee4be6897844c73504e9294eb90507f9069d63f90c9280240c14674ba11779/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/12fba9405111da6d1293dc87beaf37f2c8336a7c29ff82e1a50ce723cbe21c56/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/aea280b010dca80d1744f581a6e226922170a4e70f3769dfea85fb2c1142a232/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/c91151d43d5b7488fe126546ac61aca3fb4b0ef189954c535f8dbe8193ee7b44/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/52aac146f577a36be2f1bab3b96115338a209540e3105cc7b6c05fcb9cab79bc/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-63:{mountpoint:/var/lib/containers/storage/overlay/2610e479d192d088862362dc91cf72d2c5ae2a5cafdfd69b028dc2801a910035/merged major:0 minor:63 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/77d8af6d2b6434f7bfd397de69c1a7aaf3327101ba88e546b7b23e5778881286/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/08b47d18cb2da7659c2ff94755ce227941c3991291ebb7c899e7cd975b0fe923/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/ce79b2d77855a70feb80d5e5287676f3682bef27d3e299c9403c5d1b4ce9a4bd/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/a5accb35f7d44a414cd977da816191d989f5663e69c7e4d08e450f5ec1c86d70/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/16391b6040c1d829286bfa8c17d5c0c3bac4c56b9bd86c92bdfa216241450aca/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/4e6694db51fdd42ed866e404e9500005385507f6d7bc5da2021a2ba1a4da73be/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/5356d7452856d4706aae6d9a8ddcd6a0baaedfcdc06f51592a69205bc16d351d/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/67d68923676e71561d4b075ebd221508f75abc525278fe0d99169e7cc47e13a8/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/18353930c2ee82dfedded73a99ce2503a400fd2262f6976dd108a0010cc9c690/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 18:33:06.199410 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.199138    4372 manager.go:217] Machine: {Timestamp:2026-10-30 18:33:06.198566388 +0000 UTC m=+0.066984708 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104213 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec292fbae9a68021022edcbd4c3e98b1 SystemUUID:ec292fba-e9a6-8021-022e-dcbd4c3e98b1 BootID:709cbf4e-a181-4fb9-9b55-68c21fd38de8 Filesystems:[{Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-163 DeviceMajor:0 DeviceMinor:163 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/faeb3114-7d5c-4d94-a00c-7af6f1c175df/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/884761af-7ea5-4be1-bd94-e736df037d21/volumes/kubernetes.io~projected/kube-api-access-vq592 DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2edb2b019e72eea9d832b421121189a8b4a30a8107ddeca4cb7a3a7726b401f/userdata/shm DeviceMajor:0 DeviceMinor:59 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-63 DeviceMajor:0 DeviceMinor:63 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1836930c4aebb50d6804106a36afab66655a250c49b0a589394af2f4148e7d79/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/24897d0a-ab73-4e75-ac67-c5f5103571d0/volumes/kubernetes.io~projected/kube-api-access-4jh8p DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48282811-970d-41fb-8745-54023a7558d3/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f710dd87a10bcb0045792e084e350e9dd3a42f64ed1fcef9696675df3a5c953a/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c03044f2cb62c6599adfec04fca36cc84654335b058c6c57c251246a2265f93/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48ce8e3ccfa7d0d988f15696e21a0c8fe0cbcdb83b7d9391b4a9f7fef07197c1/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/48282811-970d-41fb-8745-54023a7558d3/volumes/kubernetes.io~projected/kube-api-access-xt8v8 DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3f0072d4-229e-452e-a533-7bf54e6f5ba4/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/24897d0a-ab73-4e75-ac67-c5f5103571d0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41efb608-1c74-4ab4-9693-53495bfc26b9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f0072d4-229e-452e-a533-7bf54e6f5ba4/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/701c1b5b-7bf7-4f11-9623-f50f2181c384/volumes/kubernetes.io~projected/kube-api-access-pjz6c DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3f0072d4-229e-452e-a533-7bf54e6f5ba4/volumes/kubernetes.io~projected/kube-api-access-w9sxz DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6c9f64e6f4e6e4ccc1bba333ff0c552523b0b069771fe7fad4592323e91cb89/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ead6483da1d900824c9541d714869bbc0a4b0b6b259dcd1dcf3f4ed05163d00/userdata/shm DeviceMajor:0 DeviceMinor:197 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c151e21-28d5-4d4a-8b95-b92d623582c5/volumes/kubernetes.io~projected/kube-api-access-5ljt9 DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6eccfeda5b686bab80f177d0880cb511c20ed2ceae94660875b10d7c7846a66c/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91317f6c-d730-44f1-a5a1-96ba9c1a6aec/volumes/kubernetes.io~projected/kube-api-access-btss5 DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/24897d0a-ab73-4e75-ac67-c5f5103571d0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd918ffa-ead9-4697-9045-da66e9c7f136/volumes/kubernetes.io~projected/kube-api-access-mwpb5 DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4ac273964b9f14facbe2259b965282a07321cf58f2e353bcdad262a4f216902/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b07f85050d7d978a0a56fd257596e801ed11cfce1425a5f08d52e1c44e9b03e/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bac59cdfb8b293760afd89f779c4e014c287848c1c496c3389615c50cb81fb61/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/382d4192-0bfc-4626-820d-5cf183296180/volumes/kubernetes.io~projected/kube-api-access-tp7fk DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ba289df1-672b-4bf2-85f6-eaffd2269631/volumes/kubernetes.io~projected/kube-api-access-wxhdz DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9f27f3523180933c4e6aa107ff07ce589d7d707440f9b74848dff1fe3249002/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/923d622a4a41829901a91e68bcb94753674b1c2685d212f58f13cb0c53c44828/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:48ce8e3ccfa7d0d MacAddress:7e:a6:17:e2:36:3f Speed:10000 Mtu:8901} {Name:9ead6483da1d900 MacAddress:66:42:a4:18:83:6c Speed:10000 Mtu:8901} {Name:bac59cdfb8b2937 MacAddress:a2:7c:ed:a8:aa:a9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e9:a9:55:53 Speed:0 Mtu:9001} {Name:br-int MacAddress:22:63:23:e2:00:c4 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:e9:a9:55:53 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:d2:cf:60:f3:2c:5e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ca:44:0f:be:0d:60 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:33:06.199536 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.199416    4372 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:33:06.199536 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.199474    4372 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:33:06.199928 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.199885    4372 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:33:06.200164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.199928    4372 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-153.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:33:06.200255 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.200169    4372 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:33:06.200255 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.200184    4372 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:33:06.200255 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.200200    4372 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:33:06.200255 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.200222    4372 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:33:06.200432 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.200418    4372 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:33:06.200699 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.200682    4372 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:33:06.200848 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.200836    4372 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:33:06.200914 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.200903    4372 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:33:06.200964 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.200927    4372 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:33:06.200964 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.200944    4372 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:33:06.200964 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.200961    4372 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:33:06.201081 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.200980    4372 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:33:06.202198 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.202178    4372 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:33:06.202198 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.202197    4372 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:33:06.202992 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.202974    4372 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:33:06.206384 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206360    4372 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:33:06.206384 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206381    4372 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:33:06.206534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206499    4372 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 18:33:06.206723 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206710    4372 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:33:06.206755 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206742    4372 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:33:06.206870 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206860    4372 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:33:06.206901 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206877    4372 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:33:06.206901 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206884    4372 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:33:06.206901 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206890    4372 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:33:06.206901 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206896    4372 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:33:06.207000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206902    4372 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:33:06.207000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206909    4372 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:33:06.207000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206915    4372 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:33:06.207000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206924    4372 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:33:06.207000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206930    4372 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:33:06.207000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206940    4372 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:33:06.207000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206950    4372 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:33:06.207000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206979    4372 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:33:06.207000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.206986    4372 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:33:06.211063 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.211035    4372 server.go:1263] "Started kubelet"
Oct 30 18:33:06.211719 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.211092    4372 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:33:06.211719 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.211207    4372 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:33:06.211719 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.211312    4372 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:33:06.211719 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.211694    4372 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:33:06.211806 ip-10-0-142-153 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:33:06.213545 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.212707    4372 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.213994    4372 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.214014    4372 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.214080    4372 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:27:28 +0000 UTC" deadline="2028-06-03 21:57:16.758084084 +0000 UTC"
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.214117    4372 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13971h24m10.543969308s"
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.214170    4372 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.214179    4372 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.214286    4372 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:33:06.215262    4372 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-153.us-east-2.compute.internal\" not found"
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.215846    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24897d0a-ab73-4e75-ac67-c5f5103571d0" volumeName="kubernetes.io/empty-dir/24897d0a-ab73-4e75-ac67-c5f5103571d0-tmp" seLinuxMountContext=""
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.215901    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f0072d4-229e-452e-a533-7bf54e6f5ba4" volumeName="kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.215923    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f0072d4-229e-452e-a533-7bf54e6f5ba4" volumeName="kubernetes.io/secret/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.215939    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41efb608-1c74-4ab4-9693-53495bfc26b9" volumeName="kubernetes.io/configmap/41efb608-1c74-4ab4-9693-53495bfc26b9-konnectivity-ca" seLinuxMountContext=""
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.215954    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c151e21-28d5-4d4a-8b95-b92d623582c5" volumeName="kubernetes.io/projected/9c151e21-28d5-4d4a-8b95-b92d623582c5-kube-api-access-5ljt9" seLinuxMountContext=""
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.215971    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba289df1-672b-4bf2-85f6-eaffd2269631" volumeName="kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.215985    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba289df1-672b-4bf2-85f6-eaffd2269631" volumeName="kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.215998    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faeb3114-7d5c-4d94-a00c-7af6f1c175df" volumeName="kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret" seLinuxMountContext=""
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216014    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24897d0a-ab73-4e75-ac67-c5f5103571d0" volumeName="kubernetes.io/projected/24897d0a-ab73-4e75-ac67-c5f5103571d0-kube-api-access-4jh8p" seLinuxMountContext=""
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216027    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="382d4192-0bfc-4626-820d-5cf183296180" volumeName="kubernetes.io/projected/382d4192-0bfc-4626-820d-5cf183296180-kube-api-access-tp7fk" seLinuxMountContext=""
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216039    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f0072d4-229e-452e-a533-7bf54e6f5ba4" volumeName="kubernetes.io/projected/3f0072d4-229e-452e-a533-7bf54e6f5ba4-kube-api-access-w9sxz" seLinuxMountContext=""
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216052    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48282811-970d-41fb-8745-54023a7558d3" volumeName="kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs" seLinuxMountContext=""
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216064    4372 factory.go:55] Registering systemd factory
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216087    4372 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216064    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="701c1b5b-7bf7-4f11-9623-f50f2181c384" volumeName="kubernetes.io/configmap/701c1b5b-7bf7-4f11-9623-f50f2181c384-iptables-alerter-script" seLinuxMountContext=""
Oct 30 18:33:06.216192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216188    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="701c1b5b-7bf7-4f11-9623-f50f2181c384" volumeName="kubernetes.io/projected/701c1b5b-7bf7-4f11-9623-f50f2181c384-kube-api-access-pjz6c" seLinuxMountContext=""
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216224    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd918ffa-ead9-4697-9045-da66e9c7f136" volumeName="kubernetes.io/projected/dd918ffa-ead9-4697-9045-da66e9c7f136-kube-api-access-mwpb5" seLinuxMountContext=""
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216242    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24897d0a-ab73-4e75-ac67-c5f5103571d0" volumeName="kubernetes.io/empty-dir/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-tuned" seLinuxMountContext=""
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216255    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="382d4192-0bfc-4626-820d-5cf183296180" volumeName="kubernetes.io/configmap/382d4192-0bfc-4626-820d-5cf183296180-serviceca" seLinuxMountContext=""
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216267    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41efb608-1c74-4ab4-9693-53495bfc26b9" volumeName="kubernetes.io/secret/41efb608-1c74-4ab4-9693-53495bfc26b9-agent-certs" seLinuxMountContext=""
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216282    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884761af-7ea5-4be1-bd94-e736df037d21" volumeName="kubernetes.io/configmap/884761af-7ea5-4be1-bd94-e736df037d21-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216294    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884761af-7ea5-4be1-bd94-e736df037d21" volumeName="kubernetes.io/configmap/884761af-7ea5-4be1-bd94-e736df037d21-multus-daemon-config" seLinuxMountContext=""
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216307    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884761af-7ea5-4be1-bd94-e736df037d21" volumeName="kubernetes.io/projected/884761af-7ea5-4be1-bd94-e736df037d21-kube-api-access-vq592" seLinuxMountContext=""
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216322    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c151e21-28d5-4d4a-8b95-b92d623582c5" volumeName="kubernetes.io/empty-dir/9c151e21-28d5-4d4a-8b95-b92d623582c5-tmp-dir" seLinuxMountContext=""
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216333    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba289df1-672b-4bf2-85f6-eaffd2269631" volumeName="kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216344    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f0072d4-229e-452e-a533-7bf54e6f5ba4" volumeName="kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-env-overrides" seLinuxMountContext=""
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216357    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f0072d4-229e-452e-a533-7bf54e6f5ba4" volumeName="kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovnkube-config" seLinuxMountContext=""
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216375    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48282811-970d-41fb-8745-54023a7558d3" volumeName="kubernetes.io/projected/48282811-970d-41fb-8745-54023a7558d3-kube-api-access-xt8v8" seLinuxMountContext=""
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216390    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91317f6c-d730-44f1-a5a1-96ba9c1a6aec" volumeName="kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5" seLinuxMountContext=""
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216438    4372 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba289df1-672b-4bf2-85f6-eaffd2269631" volumeName="kubernetes.io/projected/ba289df1-672b-4bf2-85f6-eaffd2269631-kube-api-access-wxhdz" seLinuxMountContext=""
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216459    4372 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:33:06.217499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.216469    4372 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:33:06.223353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.219201    4372 factory.go:153] Registering CRI-O factory
Oct 30 18:33:06.223353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.219231    4372 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:33:06.223353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.219310    4372 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:33:06.223353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.219332    4372 factory.go:103] Registering Raw factory
Oct 30 18:33:06.223353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.219348    4372 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:33:06.223353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.219723    4372 manager.go:319] Starting recovery of all containers
Oct 30 18:33:06.229081 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.226614    4372 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 18:33:06.229081 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.227329    4372 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:33:06.231764 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.231727    4372 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:33:06.234306 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.234269    4372 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:33:06.255392 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.255317    4372 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:33:06.257284 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.257253    4372 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:33:06.257284 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.257286    4372 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:33:06.257546 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.257314    4372 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:33:06.257546 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:33:06.257362    4372 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 18:33:06.262346 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.262299    4372 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:33:06.265749 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.265701    4372 generic.go:356] "Generic (PLEG): container finished" podID="ba289df1-672b-4bf2-85f6-eaffd2269631" containerID="0423830d87096871cc2a9a94558ff44dd379243e2527f1506efc8941ae686540" exitCode=0
Oct 30 18:33:06.265749 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.265741    4372 generic.go:356] "Generic (PLEG): container finished" podID="ba289df1-672b-4bf2-85f6-eaffd2269631" containerID="17a3c45b31b79029eec4125100883de75ad5e63af6508ca0876b7672dfefb1ac" exitCode=0
Oct 30 18:33:06.265749 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.265750    4372 generic.go:356] "Generic (PLEG): container finished" podID="ba289df1-672b-4bf2-85f6-eaffd2269631" containerID="b69f0ad16a280bce20872a34cdb0d1550beaad7485e5f118a924cae5d41cd7c2" exitCode=0
Oct 30 18:33:06.265749 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.265757    4372 generic.go:356] "Generic (PLEG): container finished" podID="ba289df1-672b-4bf2-85f6-eaffd2269631" containerID="490753a085bb889bca947b6d43d3d28e4aa2b016ad78ce0e3ac03082b29fc288" exitCode=0
Oct 30 18:33:06.266082 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.265764    4372 generic.go:356] "Generic (PLEG): container finished" podID="ba289df1-672b-4bf2-85f6-eaffd2269631" containerID="fc6354d7d79dbec86a7f7fdf22a6d86f62484779ebd157fcfaf02546e47c6326" exitCode=0
Oct 30 18:33:06.266082 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.265770    4372 generic.go:356] "Generic (PLEG): container finished" podID="ba289df1-672b-4bf2-85f6-eaffd2269631" containerID="675214b2cb619c196ce2f6fffb7d3be14bd5111b94cde32418725a077c0aaa13" exitCode=0
Oct 30 18:33:06.287272 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.287115    4372 generic.go:356] "Generic (PLEG): container finished" podID="b46186b28ca7c09f5723c9f8bbbafe8c" containerID="2b1b5ccbe0646a1b302960e70268e590b8dccf82676a0a22c00e4e2dc6e042a0" exitCode=0
Oct 30 18:33:06.294912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.294882    4372 manager.go:324] Recovery completed
Oct 30 18:33:06.313636 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.313597    4372 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:33:06.313636 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.313623    4372 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:33:06.313858 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.313653    4372 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:33:06.313891 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.313863    4372 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 18:33:06.313919 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.313880    4372 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 18:33:06.313919 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.313904    4372 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 18:33:06.313919 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.313909    4372 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 18:33:06.313919 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.313915    4372 policy_none.go:50] "Start"
Oct 30 18:33:06.314025 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.313929    4372 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:33:06.314025 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.313939    4372 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:33:06.314101 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.314089    4372 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 18:33:06.314152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.314103    4372 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 18:33:06.314152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.314110    4372 policy_none.go:44] "Start"
Oct 30 18:33:06.319145 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.319071    4372 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:33:06.322327 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.322296    4372 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:33:06.323392 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:33:06.322350    4372 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:33:06.323441 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.323401    4372 server.go:87] "Starting device plugin registration server"
Oct 30 18:33:06.323842 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.323828    4372 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:33:06.323894 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.323845    4372 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:33:06.323954 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.323933    4372 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:33:06.324097 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.324053    4372 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:33:06.324203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.324161    4372 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:33:06.324203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.324170    4372 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:33:06.324370 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.324250    4372 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:33:06.324093161Z","UUID":"9c5da5dd-f0b1-4096-9c75-33ea4759cb8d","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:33:06.325619 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:33:06.325583    4372 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:33:06.326102 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.326069    4372 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:33:06.326309 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.326117    4372 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:33:06.358219 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.358140    4372 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal"]
Oct 30 18:33:06.358750 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.358705    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal" event={"ID":"c870f7ea63961faf808ec834ef3f426f","Type":"ContainerStarted","Data":"d70eb30500b4fcc82d492c55748e0e79be2d05c2d42bb464fc33606160f3dc0f"}
Oct 30 18:33:06.358895 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.358755    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal" event={"ID":"c870f7ea63961faf808ec834ef3f426f","Type":"ContainerStarted","Data":"6eccfeda5b686bab80f177d0880cb511c20ed2ceae94660875b10d7c7846a66c"}
Oct 30 18:33:06.358895 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.358793    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal" event={"ID":"b46186b28ca7c09f5723c9f8bbbafe8c","Type":"ContainerStarted","Data":"bfb9b26634939c8f9ea96d61a4ebf78bc17647e7406601bf191ddff81366b37f"}
Oct 30 18:33:06.358895 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.358826    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal" event={"ID":"b46186b28ca7c09f5723c9f8bbbafe8c","Type":"ContainerDied","Data":"2b1b5ccbe0646a1b302960e70268e590b8dccf82676a0a22c00e4e2dc6e042a0"}
Oct 30 18:33:06.358895 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.358840    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal" event={"ID":"b46186b28ca7c09f5723c9f8bbbafe8c","Type":"ContainerStarted","Data":"d2edb2b019e72eea9d832b421121189a8b4a30a8107ddeca4cb7a3a7726b401f"}
Oct 30 18:33:06.358895 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.358888    4372 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.359111 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.358988    4372 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.372114 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.372074    4372 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:33:06.372334 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:33:06.372190    4372 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.372494 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.372476    4372 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:33:06.372553 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:33:06.372524    4372 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.424893 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.424832    4372 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:33:06.426625 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.426600    4372 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:33:06.426757 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.426640    4372 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:33:06.426757 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.426652    4372 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-153.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:33:06.426757 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.426692    4372 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.426929 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.426836    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b46186b28ca7c09f5723c9f8bbbafe8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"b46186b28ca7c09f5723c9f8bbbafe8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.426929 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.426873    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b46186b28ca7c09f5723c9f8bbbafe8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"b46186b28ca7c09f5723c9f8bbbafe8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.426929 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.426904    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c870f7ea63961faf808ec834ef3f426f-config\") pod \"kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"c870f7ea63961faf808ec834ef3f426f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.440414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.440382    4372 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.440554 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.440493    4372 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.527173 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.527067    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b46186b28ca7c09f5723c9f8bbbafe8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"b46186b28ca7c09f5723c9f8bbbafe8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.527173 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.527176    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b46186b28ca7c09f5723c9f8bbbafe8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"b46186b28ca7c09f5723c9f8bbbafe8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.527407 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.527197    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c870f7ea63961faf808ec834ef3f426f-config\") pod \"kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"c870f7ea63961faf808ec834ef3f426f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.527407 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.527205    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b46186b28ca7c09f5723c9f8bbbafe8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"b46186b28ca7c09f5723c9f8bbbafe8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.527407 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.527244    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c870f7ea63961faf808ec834ef3f426f-config\") pod \"kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"c870f7ea63961faf808ec834ef3f426f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:06.527407 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:06.527288    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b46186b28ca7c09f5723c9f8bbbafe8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal\" (UID: \"b46186b28ca7c09f5723c9f8bbbafe8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:07.201794 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.201726    4372 apiserver.go:52] "Watching apiserver"
Oct 30 18:33:07.212405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.212378    4372 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:33:07.212841 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.212818    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-2pz2f","kube-system/global-pull-secret-syncer-qmbfz","kube-system/konnectivity-agent-8vlwv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal","openshift-multus/multus-5nnbq","openshift-multus/multus-additional-cni-plugins-vlnz5","openshift-network-operator/iptables-alerter-h7gnp","openshift-dns/node-resolver-h64hp","openshift-image-registry/node-ca-9mqb5","kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-z4csl","openshift-multus/network-metrics-daemon-ngsb2","openshift-network-diagnostics/network-check-target-n52lm"]
Oct 30 18:33:07.217512 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.217475    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qwdqr\""
Oct 30 18:33:07.220623 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.220581    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:33:07.221267 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.221234    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:33:07.221440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.221412    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9qttr\""
Oct 30 18:33:07.221503 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.221453    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:33:07.221503 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.221457    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fshmx\""
Oct 30 18:33:07.221593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.221509    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:33:07.221593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.221511    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:33:07.221593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.221526    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:33:07.222360 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.222335    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:33:07.222360 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.222351    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:33:07.222360 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.222357    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:33:07.222550 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.222368    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:33:07.222550 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.222342    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:33:07.222550 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.222380    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:33:07.222873 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.222852    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:33:07.222992 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.222877    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:33:07.223274 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.223260    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:33:07.223321 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.223280    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:33:07.223849 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.223833    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-82xdm\""
Oct 30 18:33:07.223960 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.223838    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:33:07.224098 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.224082    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:33:07.224194 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.224159    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:33:07.224194 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.224167    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:33:07.224194 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.224176    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:33:07.224323 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.224199    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:33:07.224323 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.224295    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:33:07.225394 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.225365    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:33:07.225527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.225432    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:33:07.225527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.225437    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:33:07.225932 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.225915    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:33:07.225984 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.225923    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g44rk\""
Oct 30 18:33:07.225984 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.225945    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:33:07.225984 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.225923    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lc92h\""
Oct 30 18:33:07.226071 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.226020    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:33:07.226071 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.226026    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s26z4\""
Oct 30 18:33:07.226071 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.226038    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:33:07.226071 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.226045    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fgz4w\""
Oct 30 18:33:07.226194 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.226072    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z87v4\""
Oct 30 18:33:07.227169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.227117    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jhsz4\""
Oct 30 18:33:07.227233 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.227146    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-59qb5\""
Oct 30 18:33:07.227233 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.227159    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:33:07.227289 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.227230    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9fr66\""
Oct 30 18:33:07.229860 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.229830    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-ovn\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.229964 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.229875    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41efb608-1c74-4ab4-9693-53495bfc26b9-konnectivity-ca\") pod \"konnectivity-agent-8vlwv\" (UID: \"41efb608-1c74-4ab4-9693-53495bfc26b9\") " pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:33:07.229964 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.229896    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.230041 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.229964    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovn-node-metrics-cert\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.230041 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230010    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9sxz\" (UniqueName: \"kubernetes.io/projected/3f0072d4-229e-452e-a533-7bf54e6f5ba4-kube-api-access-w9sxz\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.230133 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230037    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-socket-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.230133 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230089    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/884761af-7ea5-4be1-bd94-e736df037d21-multus-daemon-config\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.230232 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230117    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-slash\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.230232 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230168    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-log-socket\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.230232 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230195    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovnkube-script-lib\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.230350 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230236    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-registration-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.230350 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230277    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41efb608-1c74-4ab4-9693-53495bfc26b9-konnectivity-ca\") pod \"konnectivity-agent-8vlwv\" (UID: \"41efb608-1c74-4ab4-9693-53495bfc26b9\") " pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:33:07.230350 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230339    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-system-cni-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.230528 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230500    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovn-node-metrics-cert\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.230528 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230521    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovnkube-script-lib\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.230846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230504    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-node-log\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.230846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230562    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/884761af-7ea5-4be1-bd94-e736df037d21-multus-daemon-config\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.230846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230584    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovnkube-config\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.230846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230629    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-device-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.230846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230679    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-sys-fs\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.230846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230708    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-cnibin\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.230846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230738    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-netns\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.230846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230760    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-conf-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.230846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230763    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-ovnkube-config\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.230846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230799    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq592\" (UniqueName: \"kubernetes.io/projected/884761af-7ea5-4be1-bd94-e736df037d21-kube-api-access-vq592\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.231334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230931    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-run-netns\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.231334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.230969    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-cni-netd\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.231334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231004    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-etc-selinux\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.231334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231038    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwpb5\" (UniqueName: \"kubernetes.io/projected/dd918ffa-ead9-4697-9045-da66e9c7f136-kube-api-access-mwpb5\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.231334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231093    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-etc-kubernetes\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.231334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231165    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-etc-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.231334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231197    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.231334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231219    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-cni-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.231334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231240    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-socket-dir-parent\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.231334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231277    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-cni-bin\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.231334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231310    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-multus-certs\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.231334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231336    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-kubelet\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.231699 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231378    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-var-lib-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.231699 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231401    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-cni-bin\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.231699 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231431    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/faeb3114-7d5c-4d94-a00c-7af6f1c175df-kubelet-config\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:33:07.231699 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231470    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/884761af-7ea5-4be1-bd94-e736df037d21-cni-binary-copy\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.231699 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231509    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-hostroot\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.231699 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231614    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.231699 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231664    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.231899 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231701    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-env-overrides\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.231899 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231717    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/884761af-7ea5-4be1-bd94-e736df037d21-cni-binary-copy\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.231899 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231738    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/faeb3114-7d5c-4d94-a00c-7af6f1c175df-dbus\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:33:07.231899 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231755    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:33:07.231899 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231890    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3f0072d4-229e-452e-a533-7bf54e6f5ba4-env-overrides\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.232144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231917    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41efb608-1c74-4ab4-9693-53495bfc26b9-agent-certs\") pod \"konnectivity-agent-8vlwv\" (UID: \"41efb608-1c74-4ab4-9693-53495bfc26b9\") " pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:33:07.232144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231941    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-kubelet\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.232144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231959    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-k8s-cni-cncf-io\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.232144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231975    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-cni-multus\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.232144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.231992    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-systemd-units\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.232144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.232030    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-systemd\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.232144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.232110    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faeb3114-7d5c-4d94-a00c-7af6f1c175df-original-pull-secret\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:33:07.232144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.232116    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41efb608-1c74-4ab4-9693-53495bfc26b9-agent-certs\") pod \"konnectivity-agent-8vlwv\" (UID: \"41efb608-1c74-4ab4-9693-53495bfc26b9\") " pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:33:07.241644 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.241608    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9sxz\" (UniqueName: \"kubernetes.io/projected/3f0072d4-229e-452e-a533-7bf54e6f5ba4-kube-api-access-w9sxz\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.241768 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.241607    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq592\" (UniqueName: \"kubernetes.io/projected/884761af-7ea5-4be1-bd94-e736df037d21-kube-api-access-vq592\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.241837 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.241818    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwpb5\" (UniqueName: \"kubernetes.io/projected/dd918ffa-ead9-4697-9045-da66e9c7f136-kube-api-access-mwpb5\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.298279 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.298231    4372 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:07.298506 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.298352    4372 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:07.312185 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.312156    4372 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:33:07.312337 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:33:07.312232    4372 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:07.314651 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.314624    4372 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:33:07.314795 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:33:07.314694    4372 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-153.us-east-2.compute.internal"
Oct 30 18:33:07.315179 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.315161    4372 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:33:07.333287 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333247    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/701c1b5b-7bf7-4f11-9623-f50f2181c384-host-slash\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:33:07.333287 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333289    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.333530 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333312    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-systemd\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.333530 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333385    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.333530 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333452    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.333530 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333444    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-lib-modules\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.333738 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333574    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-node-log\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.333738 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333618    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-device-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.333738 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333646    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-node-log\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.333738 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333685    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-sys-fs\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.333738 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333721    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt8v8\" (UniqueName: \"kubernetes.io/projected/48282811-970d-41fb-8745-54023a7558d3-kube-api-access-xt8v8\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:33:07.333738 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333733    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-device-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.333980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333750    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-cnibin\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.333980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333762    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-sys-fs\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.333980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333774    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-cni-netd\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.333980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333803    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-etc-selinux\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.333980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333824    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-cnibin\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.333980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333828    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-etc-kubernetes\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.333980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333859    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysctl-d\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.333980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333863    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-etc-kubernetes\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.333980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333889    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-cni-netd\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.333980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333905    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.333980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333911    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-etc-selinux\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.333980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333931    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-kubelet\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.333980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333947    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-var-lib-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.333980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333965    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-cni-bin\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.333980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.333981    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-kubelet\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334006    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-var-lib-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334009    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-cni-bin\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334026    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/382d4192-0bfc-4626-820d-5cf183296180-serviceca\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334060    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334086    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/faeb3114-7d5c-4d94-a00c-7af6f1c175df-dbus\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334098    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334112    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-var-lib-kubelet\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334191    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334206    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/faeb3114-7d5c-4d94-a00c-7af6f1c175df-dbus\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334242    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-systemd-units\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334278    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-systemd\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334288    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334306    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-socket-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334336    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysctl-conf\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334308    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-systemd\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334277    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-systemd-units\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334349    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-socket-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334364    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-tuned\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334394    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334437    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334475    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjz6c\" (UniqueName: \"kubernetes.io/projected/701c1b5b-7bf7-4f11-9623-f50f2181c384-kube-api-access-pjz6c\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334486    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-tuned\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.334508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334505    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp7fk\" (UniqueName: \"kubernetes.io/projected/382d4192-0bfc-4626-820d-5cf183296180-kube-api-access-tp7fk\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334537    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysconfig\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334567    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24897d0a-ab73-4e75-ac67-c5f5103571d0-tmp\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334594    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c151e21-28d5-4d4a-8b95-b92d623582c5-tmp-dir\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334690    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba289df1-672b-4bf2-85f6-eaffd2269631-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334696    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c151e21-28d5-4d4a-8b95-b92d623582c5-tmp-dir\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334733    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-slash\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334751    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24897d0a-ab73-4e75-ac67-c5f5103571d0-tmp\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334762    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-log-socket\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334791    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-registration-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334835    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-slash\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334847    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-log-socket\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334855    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c151e21-28d5-4d4a-8b95-b92d623582c5-hosts-file\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334885    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-system-cni-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334900    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/382d4192-0bfc-4626-820d-5cf183296180-serviceca\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334915    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jh8p\" (UniqueName: \"kubernetes.io/projected/24897d0a-ab73-4e75-ac67-c5f5103571d0-kube-api-access-4jh8p\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334889    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-registration-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334934    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-system-cni-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334940    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btss5\" (UniqueName: \"kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5\") pod \"network-check-target-n52lm\" (UID: \"91317f6c-d730-44f1-a5a1-96ba9c1a6aec\") " pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.334969    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxhdz\" (UniqueName: \"kubernetes.io/projected/ba289df1-672b-4bf2-85f6-eaffd2269631-kube-api-access-wxhdz\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335001    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/701c1b5b-7bf7-4f11-9623-f50f2181c384-iptables-alerter-script\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335032    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-netns\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335057    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-conf-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335102    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-run-netns\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335163    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-modprobe-d\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335194    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ljt9\" (UniqueName: \"kubernetes.io/projected/9c151e21-28d5-4d4a-8b95-b92d623582c5-kube-api-access-5ljt9\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335236    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-netns\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335244    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-etc-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335272    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335301    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-system-cni-dir\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335332    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-cni-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335362    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-socket-dir-parent\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335395    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-cni-bin\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335378    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-etc-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335410    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/701c1b5b-7bf7-4f11-9623-f50f2181c384-iptables-alerter-script\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335430    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-multus-certs\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335196    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-conf-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335472    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/faeb3114-7d5c-4d94-a00c-7af6f1c175df-kubelet-config\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335486    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-host-run-netns\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335497    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd918ffa-ead9-4697-9045-da66e9c7f136-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lcxtl\" (UID: \"dd918ffa-ead9-4697-9045-da66e9c7f136\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcxtl"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335501    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/382d4192-0bfc-4626-820d-5cf183296180-host\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:33:07.335502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335517    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-multus-certs\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335541    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-kubernetes\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335569    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-cnibin\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335584    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/faeb3114-7d5c-4d94-a00c-7af6f1c175df-kubelet-config\") pod \"global-pull-secret-syncer-qmbfz\" (UID: \"faeb3114-7d5c-4d94-a00c-7af6f1c175df\") " pod="kube-system/global-pull-secret-syncer-qmbfz"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335612    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-cni-dir\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335630    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-hostroot\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335646    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-multus-socket-dir-parent\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335676    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-hostroot\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335696    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-cni-bin\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335765    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.335712    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-openvswitch\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.336302    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-sys\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.336331    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.336357    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-kubelet\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.336394    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-k8s-cni-cncf-io\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.336400    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-kubelet\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.336415    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-cni-multus\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.336433    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-ovn\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.336445    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-run-k8s-cni-cncf-io\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.336451    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-run\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.336471    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-host\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.336481    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/884761af-7ea5-4be1-bd94-e736df037d21-host-var-lib-cni-multus\") pod \"multus-5nnbq\" (UID: \"884761af-7ea5-4be1-bd94-e736df037d21\") " pod="openshift-multus/multus-5nnbq"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.336491    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3f0072d4-229e-452e-a533-7bf54e6f5ba4-run-ovn\") pod \"ovnkube-node-2pz2f\" (UID: \"3f0072d4-229e-452e-a533-7bf54e6f5ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:07.337029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.336569    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48282811-970d-41fb-8745-54023a7558d3-metrics-certs\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:33:07.344810 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.344778    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt8v8\" (UniqueName: \"kubernetes.io/projected/48282811-970d-41fb-8745-54023a7558d3-kube-api-access-xt8v8\") pod \"network-metrics-daemon-ngsb2\" (UID: \"48282811-970d-41fb-8745-54023a7558d3\") " pod="openshift-multus/network-metrics-daemon-ngsb2"
Oct 30 18:33:07.345316 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.345292    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp7fk\" (UniqueName: \"kubernetes.io/projected/382d4192-0bfc-4626-820d-5cf183296180-kube-api-access-tp7fk\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:33:07.345430 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.345297    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjz6c\" (UniqueName: \"kubernetes.io/projected/701c1b5b-7bf7-4f11-9623-f50f2181c384-kube-api-access-pjz6c\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:33:07.345512 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.345491    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btss5\" (UniqueName: \"kubernetes.io/projected/91317f6c-d730-44f1-a5a1-96ba9c1a6aec-kube-api-access-btss5\") pod \"network-check-target-n52lm\" (UID: \"91317f6c-d730-44f1-a5a1-96ba9c1a6aec\") " pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:33:07.345923 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.345901    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxhdz\" (UniqueName: \"kubernetes.io/projected/ba289df1-672b-4bf2-85f6-eaffd2269631-kube-api-access-wxhdz\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.346207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.346186    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ljt9\" (UniqueName: \"kubernetes.io/projected/9c151e21-28d5-4d4a-8b95-b92d623582c5-kube-api-access-5ljt9\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:33:07.346261 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.346186    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jh8p\" (UniqueName: \"kubernetes.io/projected/24897d0a-ab73-4e75-ac67-c5f5103571d0-kube-api-access-4jh8p\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.437617 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437548    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-var-lib-kubelet\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.437617 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437616    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysctl-conf\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437643    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysconfig\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437661    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-var-lib-kubelet\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437673    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c151e21-28d5-4d4a-8b95-b92d623582c5-hosts-file\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437730    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-modprobe-d\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437734    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysconfig\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437752    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-system-cni-dir\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437779    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/382d4192-0bfc-4626-820d-5cf183296180-host\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437783    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c151e21-28d5-4d4a-8b95-b92d623582c5-hosts-file\") pod \"node-resolver-h64hp\" (UID: \"9c151e21-28d5-4d4a-8b95-b92d623582c5\") " pod="openshift-dns/node-resolver-h64hp"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437800    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-kubernetes\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437811    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysctl-conf\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437830    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-system-cni-dir\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437821    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-cnibin\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437862    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-kubernetes\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437843    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-cnibin\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437819    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/382d4192-0bfc-4626-820d-5cf183296180-host\") pod \"node-ca-9mqb5\" (UID: \"382d4192-0bfc-4626-820d-5cf183296180\") " pod="openshift-image-registry/node-ca-9mqb5"
Oct 30 18:33:07.437886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437880    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-modprobe-d\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437884    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-sys\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437918    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-sys\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437939    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-run\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437960    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-host\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.437978    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/701c1b5b-7bf7-4f11-9623-f50f2181c384-host-slash\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.438003    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-systemd\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.438005    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-run\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.438028    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.438036    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/701c1b5b-7bf7-4f11-9623-f50f2181c384-host-slash\") pod \"iptables-alerter-h7gnp\" (UID: \"701c1b5b-7bf7-4f11-9623-f50f2181c384\") " pod="openshift-network-operator/iptables-alerter-h7gnp"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.438054    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-systemd\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.438055    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-lib-modules\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.438057    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-host\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.438080    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba289df1-672b-4bf2-85f6-eaffd2269631-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlnz5\" (UID: \"ba289df1-672b-4bf2-85f6-eaffd2269631\") " pod="openshift-multus/multus-additional-cni-plugins-vlnz5"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.438100    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysctl-d\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.438146    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-lib-modules\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.438534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.438234    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24897d0a-ab73-4e75-ac67-c5f5103571d0-etc-sysctl-d\") pod \"tuned-z4csl\" (UID: \"24897d0a-ab73-4e75-ac67-c5f5103571d0\") " pod="openshift-cluster-node-tuning-operator/tuned-z4csl"
Oct 30 18:33:07.842873 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:07.842804    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:33:09.146796 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:09.146746    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:12.625832 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:12.625784    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:12.626419 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:12.626021    4372 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:33:15.415097 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:15.415059    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8vlwv"
Oct 30 18:33:17.349737 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.349666    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-xn6pz-hnbfq"]
Oct 30 18:33:17.353309 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.353276    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:33:17.364066 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.364030    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:33:17.364232 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.364084    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:33:17.369602 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.369567    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-g4tgh\""
Oct 30 18:33:17.372191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.371433    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 18:33:17.372191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.371617    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 18:33:17.372191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.372112    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xn6pz-hnbfq"]
Oct 30 18:33:17.396732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.396673    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57a61f9-b181-4622-8e40-68402dde03fc-trusted-ca-bundle\") pod \"periodic-gathering-xn6pz-hnbfq\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") " pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:33:17.396969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.396821    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c57a61f9-b181-4622-8e40-68402dde03fc-archives-path\") pod \"periodic-gathering-xn6pz-hnbfq\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") " pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:33:17.396969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.396891    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57a61f9-b181-4622-8e40-68402dde03fc-service-ca-bundle\") pod \"periodic-gathering-xn6pz-hnbfq\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") " pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:33:17.396969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.396923    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmtss\" (UniqueName: \"kubernetes.io/projected/c57a61f9-b181-4622-8e40-68402dde03fc-kube-api-access-bmtss\") pod \"periodic-gathering-xn6pz-hnbfq\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") " pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:33:17.401589 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.401556    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-fx7q7"]
Oct 30 18:33:17.404693 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.404662    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-fx7q7"
Oct 30 18:33:17.406648 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.406615    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-89d5c458-w78vx"]
Oct 30 18:33:17.410625 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.410594    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.425712 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.425678    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 18:33:17.425919 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.425816    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 18:33:17.426209 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.426185    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 18:33:17.426353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.426215    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 18:33:17.430378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.430333    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-89d5c458-w78vx"]
Oct 30 18:33:17.430720 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.430703    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-fx7q7"]
Oct 30 18:33:17.441464 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.441421    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vgjw8"]
Oct 30 18:33:17.444598 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.444552    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vgjw8"
Oct 30 18:33:17.446940 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.446904    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 18:33:17.463332 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.463285    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vgjw8"]
Oct 30 18:33:17.464618 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.464587    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:33:17.464788 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.464765    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-bb9c4\""
Oct 30 18:33:17.464844 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.464833    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5gd7g\""
Oct 30 18:33:17.464890 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.464834    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:33:17.464890 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.464848    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 18:33:17.464963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.464899    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:33:17.464963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.464938    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6twlp\""
Oct 30 18:33:17.465786 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.465763    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 18:33:17.487157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.487085    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2t9nh"]
Oct 30 18:33:17.490438 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.490413    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.493927 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.493893    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h68bb"]
Oct 30 18:33:17.497420 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.497377    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/795ba6e5-0a07-487c-b349-772ac223ffab-console-serving-cert\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.497420 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.497421    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-console-config\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.497699 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.497540    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d02d01c9-1b8c-4632-866c-3fecfda75c4a-tmp\") pod \"downloads-6dd789c8b7-fx7q7\" (UID: \"d02d01c9-1b8c-4632-866c-3fecfda75c4a\") " pod="openshift-console/downloads-6dd789c8b7-fx7q7"
Oct 30 18:33:17.497699 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.497590    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9q9d\" (UniqueName: \"kubernetes.io/projected/dae5359a-4e6c-4a5d-a688-aff0f02cb240-kube-api-access-p9q9d\") pod \"ingress-canary-vgjw8\" (UID: \"dae5359a-4e6c-4a5d-a688-aff0f02cb240\") " pod="openshift-ingress-canary/ingress-canary-vgjw8"
Oct 30 18:33:17.497777 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.497703    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/795ba6e5-0a07-487c-b349-772ac223ffab-console-oauth-config\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.497814 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.497788    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c57a61f9-b181-4622-8e40-68402dde03fc-archives-path\") pod \"periodic-gathering-xn6pz-hnbfq\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") " pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:33:17.497854 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.497818    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dae5359a-4e6c-4a5d-a688-aff0f02cb240-cert\") pod \"ingress-canary-vgjw8\" (UID: \"dae5359a-4e6c-4a5d-a688-aff0f02cb240\") " pod="openshift-ingress-canary/ingress-canary-vgjw8"
Oct 30 18:33:17.497854 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.497842    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57a61f9-b181-4622-8e40-68402dde03fc-service-ca-bundle\") pod \"periodic-gathering-xn6pz-hnbfq\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") " pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:33:17.497946 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.497858    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmtss\" (UniqueName: \"kubernetes.io/projected/c57a61f9-b181-4622-8e40-68402dde03fc-kube-api-access-bmtss\") pod \"periodic-gathering-xn6pz-hnbfq\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") " pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:33:17.497946 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.497875    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/795ba6e5-0a07-487c-b349-772ac223ffab-tmp\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.498040 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.498013    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-oauth-serving-cert\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.498094 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.498051    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spz5h\" (UniqueName: \"kubernetes.io/projected/d02d01c9-1b8c-4632-866c-3fecfda75c4a-kube-api-access-spz5h\") pod \"downloads-6dd789c8b7-fx7q7\" (UID: \"d02d01c9-1b8c-4632-866c-3fecfda75c4a\") " pod="openshift-console/downloads-6dd789c8b7-fx7q7"
Oct 30 18:33:17.498189 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.498117    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57a61f9-b181-4622-8e40-68402dde03fc-trusted-ca-bundle\") pod \"periodic-gathering-xn6pz-hnbfq\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") " pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:33:17.498267 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.498190    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-service-ca\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.498267 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.498237    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8fg\" (UniqueName: \"kubernetes.io/projected/795ba6e5-0a07-487c-b349-772ac223ffab-kube-api-access-sx8fg\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.498378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.498333    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c57a61f9-b181-4622-8e40-68402dde03fc-archives-path\") pod \"periodic-gathering-xn6pz-hnbfq\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") " pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:33:17.498541 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.498523    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2j794"]
Oct 30 18:33:17.498678 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.498659    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h68bb"
Oct 30 18:33:17.498752 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.498735    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57a61f9-b181-4622-8e40-68402dde03fc-service-ca-bundle\") pod \"periodic-gathering-xn6pz-hnbfq\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") " pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:33:17.499541 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.499300    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57a61f9-b181-4622-8e40-68402dde03fc-trusted-ca-bundle\") pod \"periodic-gathering-xn6pz-hnbfq\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") " pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:33:17.500363 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.500340    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:33:17.502328 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.502303    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:17.507072 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.507039    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2t9nh"]
Oct 30 18:33:17.511919 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.511870    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h68bb"]
Oct 30 18:33:17.525738 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.525691    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:33:17.527198 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.527168    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2j794"]
Oct 30 18:33:17.531066 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.531029    4372 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:33:17.535821 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.535792    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmtss\" (UniqueName: \"kubernetes.io/projected/c57a61f9-b181-4622-8e40-68402dde03fc-kube-api-access-bmtss\") pod \"periodic-gathering-xn6pz-hnbfq\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") " pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:33:17.565565 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.565527    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:33:17.565565 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.565527    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mzdhg\""
Oct 30 18:33:17.565834 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.565577    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 18:33:17.565834 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.565533    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pg8bd\""
Oct 30 18:33:17.565834 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.565626    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:33:17.565834 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.565647    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bqrtm\""
Oct 30 18:33:17.593882 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.593843    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fd56c4b66-72fh4"]
Oct 30 18:33:17.597541 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.597510    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.599224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599189    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8fg\" (UniqueName: \"kubernetes.io/projected/795ba6e5-0a07-487c-b349-772ac223ffab-kube-api-access-sx8fg\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.599412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599236    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/795ba6e5-0a07-487c-b349-772ac223ffab-console-serving-cert\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.599412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599257    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-console-config\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.599412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599274    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d02d01c9-1b8c-4632-866c-3fecfda75c4a-tmp\") pod \"downloads-6dd789c8b7-fx7q7\" (UID: \"d02d01c9-1b8c-4632-866c-3fecfda75c4a\") " pod="openshift-console/downloads-6dd789c8b7-fx7q7"
Oct 30 18:33:17.599412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599294    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76637ad2-0d7d-4c03-a808-13e0125424a2-crio-socket\") pod \"insights-runtime-extractor-2t9nh\" (UID: \"76637ad2-0d7d-4c03-a808-13e0125424a2\") " pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.599412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599315    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76637ad2-0d7d-4c03-a808-13e0125424a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2t9nh\" (UID: \"76637ad2-0d7d-4c03-a808-13e0125424a2\") " pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.599412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599381    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9q9d\" (UniqueName: \"kubernetes.io/projected/dae5359a-4e6c-4a5d-a688-aff0f02cb240-kube-api-access-p9q9d\") pod \"ingress-canary-vgjw8\" (UID: \"dae5359a-4e6c-4a5d-a688-aff0f02cb240\") " pod="openshift-ingress-canary/ingress-canary-vgjw8"
Oct 30 18:33:17.599412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599401    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74d61d4b-1a8f-4256-aae6-c51763689c92-metrics-tls\") pod \"dns-default-2j794\" (UID: \"74d61d4b-1a8f-4256-aae6-c51763689c92\") " pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:17.599640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599427    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/795ba6e5-0a07-487c-b349-772ac223ffab-console-oauth-config\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.599640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599447    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76637ad2-0d7d-4c03-a808-13e0125424a2-data-volume\") pod \"insights-runtime-extractor-2t9nh\" (UID: \"76637ad2-0d7d-4c03-a808-13e0125424a2\") " pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.599640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599463    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79ftb\" (UniqueName: \"kubernetes.io/projected/74d61d4b-1a8f-4256-aae6-c51763689c92-kube-api-access-79ftb\") pod \"dns-default-2j794\" (UID: \"74d61d4b-1a8f-4256-aae6-c51763689c92\") " pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:17.599640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599493    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dae5359a-4e6c-4a5d-a688-aff0f02cb240-cert\") pod \"ingress-canary-vgjw8\" (UID: \"dae5359a-4e6c-4a5d-a688-aff0f02cb240\") " pod="openshift-ingress-canary/ingress-canary-vgjw8"
Oct 30 18:33:17.599640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599518    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74d61d4b-1a8f-4256-aae6-c51763689c92-tmp-dir\") pod \"dns-default-2j794\" (UID: \"74d61d4b-1a8f-4256-aae6-c51763689c92\") " pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:17.599640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599535    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/795ba6e5-0a07-487c-b349-772ac223ffab-tmp\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.599640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599554    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-oauth-serving-cert\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.599640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599572    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spz5h\" (UniqueName: \"kubernetes.io/projected/d02d01c9-1b8c-4632-866c-3fecfda75c4a-kube-api-access-spz5h\") pod \"downloads-6dd789c8b7-fx7q7\" (UID: \"d02d01c9-1b8c-4632-866c-3fecfda75c4a\") " pod="openshift-console/downloads-6dd789c8b7-fx7q7"
Oct 30 18:33:17.599640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599595    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e07c1ad1-f5e7-49b0-b7b7-078cb19aa615-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-h68bb\" (UID: \"e07c1ad1-f5e7-49b0-b7b7-078cb19aa615\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h68bb"
Oct 30 18:33:17.600005 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599635    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74d61d4b-1a8f-4256-aae6-c51763689c92-config-volume\") pod \"dns-default-2j794\" (UID: \"74d61d4b-1a8f-4256-aae6-c51763689c92\") " pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:17.600005 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599743    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d02d01c9-1b8c-4632-866c-3fecfda75c4a-tmp\") pod \"downloads-6dd789c8b7-fx7q7\" (UID: \"d02d01c9-1b8c-4632-866c-3fecfda75c4a\") " pod="openshift-console/downloads-6dd789c8b7-fx7q7"
Oct 30 18:33:17.600005 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599846    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-service-ca\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.600005 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.599909    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76637ad2-0d7d-4c03-a808-13e0125424a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2t9nh\" (UID: \"76637ad2-0d7d-4c03-a808-13e0125424a2\") " pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.600398 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.600327    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-console-config\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.600454 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.600412    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/795ba6e5-0a07-487c-b349-772ac223ffab-tmp\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.600591 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.600560    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdqp6\" (UniqueName: \"kubernetes.io/projected/76637ad2-0d7d-4c03-a808-13e0125424a2-kube-api-access-gdqp6\") pod \"insights-runtime-extractor-2t9nh\" (UID: \"76637ad2-0d7d-4c03-a808-13e0125424a2\") " pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.601259 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.601230    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-service-ca\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.601399 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.601230    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-oauth-serving-cert\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.602831 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.602798    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/795ba6e5-0a07-487c-b349-772ac223ffab-console-oauth-config\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.602987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.602927    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/795ba6e5-0a07-487c-b349-772ac223ffab-console-serving-cert\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.603745 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.603721    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dae5359a-4e6c-4a5d-a688-aff0f02cb240-cert\") pod \"ingress-canary-vgjw8\" (UID: \"dae5359a-4e6c-4a5d-a688-aff0f02cb240\") " pod="openshift-ingress-canary/ingress-canary-vgjw8"
Oct 30 18:33:17.613206 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.613163    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 18:33:17.617290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.617238    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spz5h\" (UniqueName: \"kubernetes.io/projected/d02d01c9-1b8c-4632-866c-3fecfda75c4a-kube-api-access-spz5h\") pod \"downloads-6dd789c8b7-fx7q7\" (UID: \"d02d01c9-1b8c-4632-866c-3fecfda75c4a\") " pod="openshift-console/downloads-6dd789c8b7-fx7q7"
Oct 30 18:33:17.617489 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.617396    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8fg\" (UniqueName: \"kubernetes.io/projected/795ba6e5-0a07-487c-b349-772ac223ffab-kube-api-access-sx8fg\") pod \"console-89d5c458-w78vx\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") " pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.618215 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.618192    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9q9d\" (UniqueName: \"kubernetes.io/projected/dae5359a-4e6c-4a5d-a688-aff0f02cb240-kube-api-access-p9q9d\") pod \"ingress-canary-vgjw8\" (UID: \"dae5359a-4e6c-4a5d-a688-aff0f02cb240\") " pod="openshift-ingress-canary/ingress-canary-vgjw8"
Oct 30 18:33:17.621861 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.621823    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fd56c4b66-72fh4"]
Oct 30 18:33:17.664799 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.664762    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 18:33:17.665047 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.664901    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2fw8l\""
Oct 30 18:33:17.665047 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.665019    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 18:33:17.665156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.665103    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 18:33:17.667017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.666986    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:33:17.702187 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.702097    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa15387d-68c0-4a12-a921-313ba8743b0a-registry-tls\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.702511 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.702286    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e07c1ad1-f5e7-49b0-b7b7-078cb19aa615-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-h68bb\" (UID: \"e07c1ad1-f5e7-49b0-b7b7-078cb19aa615\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h68bb"
Oct 30 18:33:17.702511 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.702344    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74d61d4b-1a8f-4256-aae6-c51763689c92-config-volume\") pod \"dns-default-2j794\" (UID: \"74d61d4b-1a8f-4256-aae6-c51763689c92\") " pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:17.702511 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.702384    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76637ad2-0d7d-4c03-a808-13e0125424a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2t9nh\" (UID: \"76637ad2-0d7d-4c03-a808-13e0125424a2\") " pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.702511 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.702414    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdqp6\" (UniqueName: \"kubernetes.io/projected/76637ad2-0d7d-4c03-a808-13e0125424a2-kube-api-access-gdqp6\") pod \"insights-runtime-extractor-2t9nh\" (UID: \"76637ad2-0d7d-4c03-a808-13e0125424a2\") " pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.702511 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.702452    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa15387d-68c0-4a12-a921-313ba8743b0a-bound-sa-token\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.702511 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.702481    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74d61d4b-1a8f-4256-aae6-c51763689c92-metrics-tls\") pod \"dns-default-2j794\" (UID: \"74d61d4b-1a8f-4256-aae6-c51763689c92\") " pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:17.703015 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.702978    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76637ad2-0d7d-4c03-a808-13e0125424a2-crio-socket\") pod \"insights-runtime-extractor-2t9nh\" (UID: \"76637ad2-0d7d-4c03-a808-13e0125424a2\") " pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.703113 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.703032    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76637ad2-0d7d-4c03-a808-13e0125424a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2t9nh\" (UID: \"76637ad2-0d7d-4c03-a808-13e0125424a2\") " pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.703113 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.703069    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfsld\" (UniqueName: \"kubernetes.io/projected/fa15387d-68c0-4a12-a921-313ba8743b0a-kube-api-access-jfsld\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.703113 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.703092    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76637ad2-0d7d-4c03-a808-13e0125424a2-data-volume\") pod \"insights-runtime-extractor-2t9nh\" (UID: \"76637ad2-0d7d-4c03-a808-13e0125424a2\") " pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.703293 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.703140    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74d61d4b-1a8f-4256-aae6-c51763689c92-tmp-dir\") pod \"dns-default-2j794\" (UID: \"74d61d4b-1a8f-4256-aae6-c51763689c92\") " pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:17.703293 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.703199    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa15387d-68c0-4a12-a921-313ba8743b0a-ca-trust-extracted\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.703293 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.703219    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74d61d4b-1a8f-4256-aae6-c51763689c92-config-volume\") pod \"dns-default-2j794\" (UID: \"74d61d4b-1a8f-4256-aae6-c51763689c92\") " pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:17.703293 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.703230    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa15387d-68c0-4a12-a921-313ba8743b0a-image-registry-private-configuration\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.703653 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.703373    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa15387d-68c0-4a12-a921-313ba8743b0a-registry-certificates\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.703653 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.703403    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79ftb\" (UniqueName: \"kubernetes.io/projected/74d61d4b-1a8f-4256-aae6-c51763689c92-kube-api-access-79ftb\") pod \"dns-default-2j794\" (UID: \"74d61d4b-1a8f-4256-aae6-c51763689c92\") " pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:17.703653 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.703439    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa15387d-68c0-4a12-a921-313ba8743b0a-installation-pull-secrets\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.703653 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.703470    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa15387d-68c0-4a12-a921-313ba8743b0a-trusted-ca\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.703653 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.703539    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74d61d4b-1a8f-4256-aae6-c51763689c92-tmp-dir\") pod \"dns-default-2j794\" (UID: \"74d61d4b-1a8f-4256-aae6-c51763689c92\") " pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:17.703907 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.703753    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76637ad2-0d7d-4c03-a808-13e0125424a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2t9nh\" (UID: \"76637ad2-0d7d-4c03-a808-13e0125424a2\") " pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.703907 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.703833    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76637ad2-0d7d-4c03-a808-13e0125424a2-crio-socket\") pod \"insights-runtime-extractor-2t9nh\" (UID: \"76637ad2-0d7d-4c03-a808-13e0125424a2\") " pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.704162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.704108    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76637ad2-0d7d-4c03-a808-13e0125424a2-data-volume\") pod \"insights-runtime-extractor-2t9nh\" (UID: \"76637ad2-0d7d-4c03-a808-13e0125424a2\") " pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.705628 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.705601    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74d61d4b-1a8f-4256-aae6-c51763689c92-metrics-tls\") pod \"dns-default-2j794\" (UID: \"74d61d4b-1a8f-4256-aae6-c51763689c92\") " pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:17.705768 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.705681    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e07c1ad1-f5e7-49b0-b7b7-078cb19aa615-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-h68bb\" (UID: \"e07c1ad1-f5e7-49b0-b7b7-078cb19aa615\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h68bb"
Oct 30 18:33:17.705768 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.705725    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76637ad2-0d7d-4c03-a808-13e0125424a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2t9nh\" (UID: \"76637ad2-0d7d-4c03-a808-13e0125424a2\") " pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.717183 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.717111    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdqp6\" (UniqueName: \"kubernetes.io/projected/76637ad2-0d7d-4c03-a808-13e0125424a2-kube-api-access-gdqp6\") pod \"insights-runtime-extractor-2t9nh\" (UID: \"76637ad2-0d7d-4c03-a808-13e0125424a2\") " pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.717183 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.717181    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79ftb\" (UniqueName: \"kubernetes.io/projected/74d61d4b-1a8f-4256-aae6-c51763689c92-kube-api-access-79ftb\") pod \"dns-default-2j794\" (UID: \"74d61d4b-1a8f-4256-aae6-c51763689c92\") " pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:17.719010 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.718983    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-fx7q7"
Oct 30 18:33:17.724636 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.724593    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:17.759162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.757238    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vgjw8"
Oct 30 18:33:17.804683 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.804639    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa15387d-68c0-4a12-a921-313ba8743b0a-registry-certificates\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.804683 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.804689    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa15387d-68c0-4a12-a921-313ba8743b0a-installation-pull-secrets\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.805260 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.804712    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa15387d-68c0-4a12-a921-313ba8743b0a-trusted-ca\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.805260 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.804741    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa15387d-68c0-4a12-a921-313ba8743b0a-registry-tls\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.805260 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.804766    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa15387d-68c0-4a12-a921-313ba8743b0a-bound-sa-token\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.805260 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.804791    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfsld\" (UniqueName: \"kubernetes.io/projected/fa15387d-68c0-4a12-a921-313ba8743b0a-kube-api-access-jfsld\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.805260 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.804828    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa15387d-68c0-4a12-a921-313ba8743b0a-ca-trust-extracted\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.805260 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.804847    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa15387d-68c0-4a12-a921-313ba8743b0a-image-registry-private-configuration\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.807305 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.806614    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa15387d-68c0-4a12-a921-313ba8743b0a-registry-certificates\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.807305 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.807246    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa15387d-68c0-4a12-a921-313ba8743b0a-ca-trust-extracted\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.808240 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.808192    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa15387d-68c0-4a12-a921-313ba8743b0a-trusted-ca\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.810414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.809879    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa15387d-68c0-4a12-a921-313ba8743b0a-registry-tls\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.810414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.810059    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2t9nh"
Oct 30 18:33:17.814164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.811281    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa15387d-68c0-4a12-a921-313ba8743b0a-installation-pull-secrets\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.814164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.812554    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa15387d-68c0-4a12-a921-313ba8743b0a-image-registry-private-configuration\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.817596 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.817554    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h68bb"
Oct 30 18:33:17.820810 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.820776    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa15387d-68c0-4a12-a921-313ba8743b0a-bound-sa-token\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.820991 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.820944    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfsld\" (UniqueName: \"kubernetes.io/projected/fa15387d-68c0-4a12-a921-313ba8743b0a-kube-api-access-jfsld\") pod \"image-registry-fd56c4b66-72fh4\" (UID: \"fa15387d-68c0-4a12-a921-313ba8743b0a\") " pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.825391 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.825308    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:17.938790 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.938033    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:17.951733 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.951694    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xn6pz-hnbfq"]
Oct 30 18:33:17.973882 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:17.973700    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57a61f9_b181_4622_8e40_68402dde03fc.slice/crio-dbd9ce47886a33edff0ea00ff25c6762240094b634c005f35492dc4b5f46a497 WatchSource:0}: Error finding container dbd9ce47886a33edff0ea00ff25c6762240094b634c005f35492dc4b5f46a497: Status 404 returned error can't find the container with id dbd9ce47886a33edff0ea00ff25c6762240094b634c005f35492dc4b5f46a497
Oct 30 18:33:17.983488 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:17.981803    4372 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:33:18.027895 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.027854    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-fx7q7"]
Oct 30 18:33:18.029972 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.029935    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-89d5c458-w78vx"]
Oct 30 18:33:18.035465 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:18.035426    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02d01c9_1b8c_4632_866c_3fecfda75c4a.slice/crio-79972e6ed294305098180b68270411c9b7d8c47c3f4b3e3c5d1fa7420b8d3d91 WatchSource:0}: Error finding container 79972e6ed294305098180b68270411c9b7d8c47c3f4b3e3c5d1fa7420b8d3d91: Status 404 returned error can't find the container with id 79972e6ed294305098180b68270411c9b7d8c47c3f4b3e3c5d1fa7420b8d3d91
Oct 30 18:33:18.036211 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:18.036191    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795ba6e5_0a07_487c_b349_772ac223ffab.slice/crio-d168081812e8d5f830c1f95cdad76b1affb455795c5d8e50fa0bab2b0374cf8a WatchSource:0}: Error finding container d168081812e8d5f830c1f95cdad76b1affb455795c5d8e50fa0bab2b0374cf8a: Status 404 returned error can't find the container with id d168081812e8d5f830c1f95cdad76b1affb455795c5d8e50fa0bab2b0374cf8a
Oct 30 18:33:18.088926 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.088892    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vgjw8"]
Oct 30 18:33:18.092632 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:18.092592    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae5359a_4e6c_4a5d_a688_aff0f02cb240.slice/crio-b8e64b4a16ee6c7b3bb2d8923ab23d6386a5f1e0155f455eeee90c0c5a9d90f1 WatchSource:0}: Error finding container b8e64b4a16ee6c7b3bb2d8923ab23d6386a5f1e0155f455eeee90c0c5a9d90f1: Status 404 returned error can't find the container with id b8e64b4a16ee6c7b3bb2d8923ab23d6386a5f1e0155f455eeee90c0c5a9d90f1
Oct 30 18:33:18.160661 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.160557    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2t9nh"]
Oct 30 18:33:18.163279 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.163252    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2j794"]
Oct 30 18:33:18.165597 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:18.165566    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76637ad2_0d7d_4c03_a808_13e0125424a2.slice/crio-25e98675c4d5c5aa89e229c103e4e20e044d44adc6225f0cb9deac445f07bdcc WatchSource:0}: Error finding container 25e98675c4d5c5aa89e229c103e4e20e044d44adc6225f0cb9deac445f07bdcc: Status 404 returned error can't find the container with id 25e98675c4d5c5aa89e229c103e4e20e044d44adc6225f0cb9deac445f07bdcc
Oct 30 18:33:18.166424 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:18.166399    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d61d4b_1a8f_4256_aae6_c51763689c92.slice/crio-31f9ab61f1fc607d8e09f24b685d6c18be1f6caf847e8a388c46819dd42145cc WatchSource:0}: Error finding container 31f9ab61f1fc607d8e09f24b685d6c18be1f6caf847e8a388c46819dd42145cc: Status 404 returned error can't find the container with id 31f9ab61f1fc607d8e09f24b685d6c18be1f6caf847e8a388c46819dd42145cc
Oct 30 18:33:18.172420 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.172385    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h68bb"]
Oct 30 18:33:18.176723 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:18.176675    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07c1ad1_f5e7_49b0_b7b7_078cb19aa615.slice/crio-7160171c7c72c62a5e812cdf43b46045bba2baf77239be95a122c02011d71746 WatchSource:0}: Error finding container 7160171c7c72c62a5e812cdf43b46045bba2baf77239be95a122c02011d71746: Status 404 returned error can't find the container with id 7160171c7c72c62a5e812cdf43b46045bba2baf77239be95a122c02011d71746
Oct 30 18:33:18.253593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.253550    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fd56c4b66-72fh4"]
Oct 30 18:33:18.259018 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:18.257479    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa15387d_68c0_4a12_a921_313ba8743b0a.slice/crio-adeaa3a567c71c5c1ae8e9e189b0c2758c1237372434240ee155a6e415b57d5c WatchSource:0}: Error finding container adeaa3a567c71c5c1ae8e9e189b0c2758c1237372434240ee155a6e415b57d5c: Status 404 returned error can't find the container with id adeaa3a567c71c5c1ae8e9e189b0c2758c1237372434240ee155a6e415b57d5c
Oct 30 18:33:18.330469 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.330415    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h68bb" event={"ID":"e07c1ad1-f5e7-49b0-b7b7-078cb19aa615","Type":"ContainerStarted","Data":"7160171c7c72c62a5e812cdf43b46045bba2baf77239be95a122c02011d71746"}
Oct 30 18:33:18.332115 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.332070    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2t9nh" event={"ID":"76637ad2-0d7d-4c03-a808-13e0125424a2","Type":"ContainerStarted","Data":"fcf8018a456eeec047214a63d5eb17a3bfbe2eee94e8892ccf131653f4960430"}
Oct 30 18:33:18.332344 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.332174    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2t9nh" event={"ID":"76637ad2-0d7d-4c03-a808-13e0125424a2","Type":"ContainerStarted","Data":"25e98675c4d5c5aa89e229c103e4e20e044d44adc6225f0cb9deac445f07bdcc"}
Oct 30 18:33:18.333471 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.333440    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-89d5c458-w78vx" event={"ID":"795ba6e5-0a07-487c-b349-772ac223ffab","Type":"ContainerStarted","Data":"d168081812e8d5f830c1f95cdad76b1affb455795c5d8e50fa0bab2b0374cf8a"}
Oct 30 18:33:18.334751 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.334720    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-fx7q7" event={"ID":"d02d01c9-1b8c-4632-866c-3fecfda75c4a","Type":"ContainerStarted","Data":"79972e6ed294305098180b68270411c9b7d8c47c3f4b3e3c5d1fa7420b8d3d91"}
Oct 30 18:33:18.336036 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.336002    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xn6pz-hnbfq" event={"ID":"c57a61f9-b181-4622-8e40-68402dde03fc","Type":"ContainerStarted","Data":"dbd9ce47886a33edff0ea00ff25c6762240094b634c005f35492dc4b5f46a497"}
Oct 30 18:33:18.337747 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.337717    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fd56c4b66-72fh4" event={"ID":"fa15387d-68c0-4a12-a921-313ba8743b0a","Type":"ContainerStarted","Data":"21bb729f928c936b086fa721c62d32828b5645c1dff119fdf7bca195abccea32"}
Oct 30 18:33:18.337881 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.337758    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fd56c4b66-72fh4" event={"ID":"fa15387d-68c0-4a12-a921-313ba8743b0a","Type":"ContainerStarted","Data":"adeaa3a567c71c5c1ae8e9e189b0c2758c1237372434240ee155a6e415b57d5c"}
Oct 30 18:33:18.337881 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.337823    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:18.339084 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.339057    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2j794" event={"ID":"74d61d4b-1a8f-4256-aae6-c51763689c92","Type":"ContainerStarted","Data":"31f9ab61f1fc607d8e09f24b685d6c18be1f6caf847e8a388c46819dd42145cc"}
Oct 30 18:33:18.340206 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.340179    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vgjw8" event={"ID":"dae5359a-4e6c-4a5d-a688-aff0f02cb240","Type":"ContainerStarted","Data":"b8e64b4a16ee6c7b3bb2d8923ab23d6386a5f1e0155f455eeee90c0c5a9d90f1"}
Oct 30 18:33:18.392948 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:18.392862    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fd56c4b66-72fh4" podStartSLOduration=1.392835819 podStartE2EDuration="1.392835819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:18.392628536 +0000 UTC m=+12.261046879" watchObservedRunningTime="2026-10-30 18:33:18.392835819 +0000 UTC m=+12.261254152"
Oct 30 18:33:19.355967 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:19.355914    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2t9nh" event={"ID":"76637ad2-0d7d-4c03-a808-13e0125424a2","Type":"ContainerStarted","Data":"a51ffe626a11d9e187d26e19a4be349141aaece2aa4b9512dca2ae0c2c0147c0"}
Oct 30 18:33:20.366534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:20.366488    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h68bb" event={"ID":"e07c1ad1-f5e7-49b0-b7b7-078cb19aa615","Type":"ContainerStarted","Data":"4d18aed9abeca30938b0626be755bc784ae93e2fbf33b1f4580a2ef7a9df4480"}
Oct 30 18:33:20.367839 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:20.367745    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h68bb"
Oct 30 18:33:20.374891 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:20.374841    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h68bb"
Oct 30 18:33:20.388292 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:20.388020    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h68bb" podStartSLOduration=1.425948333 podStartE2EDuration="3.387993684s" totalImagesPullingTime="1.962045351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:20.38694342 +0000 UTC m=+14.255361752" watchObservedRunningTime="2026-10-30 18:33:20.387993684 +0000 UTC m=+14.256411997"
Oct 30 18:33:21.386630 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.384964    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"]
Oct 30 18:33:21.390509 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.389218    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"
Oct 30 18:33:21.401579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.400824    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 18:33:21.402759 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.401966    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"]
Oct 30 18:33:21.402831 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.402790    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7rlr8\""
Oct 30 18:33:21.403033 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.403003    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 18:33:21.406093 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.406029    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 18:33:21.406332 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.406262    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:33:21.406332 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.406317    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:33:21.419584 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.419525    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8b88fb85b-6bwlm"]
Oct 30 18:33:21.423249 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.423193    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.439209 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.439060    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b88fb85b-6bwlm"]
Oct 30 18:33:21.446297 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.441401    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5trmt\" (UniqueName: \"kubernetes.io/projected/c80b2cb2-7c3f-42dd-ba8f-017131416961-kube-api-access-5trmt\") pod \"prometheus-operator-9cd89b74c-kk6bc\" (UID: \"c80b2cb2-7c3f-42dd-ba8f-017131416961\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"
Oct 30 18:33:21.446619 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.446590    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c80b2cb2-7c3f-42dd-ba8f-017131416961-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kk6bc\" (UID: \"c80b2cb2-7c3f-42dd-ba8f-017131416961\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"
Oct 30 18:33:21.446714 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.446688    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c80b2cb2-7c3f-42dd-ba8f-017131416961-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kk6bc\" (UID: \"c80b2cb2-7c3f-42dd-ba8f-017131416961\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"
Oct 30 18:33:21.446827 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.446807    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c80b2cb2-7c3f-42dd-ba8f-017131416961-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kk6bc\" (UID: \"c80b2cb2-7c3f-42dd-ba8f-017131416961\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"
Oct 30 18:33:21.509734 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.509678    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 18:33:21.548187 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.548112    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5trmt\" (UniqueName: \"kubernetes.io/projected/c80b2cb2-7c3f-42dd-ba8f-017131416961-kube-api-access-5trmt\") pod \"prometheus-operator-9cd89b74c-kk6bc\" (UID: \"c80b2cb2-7c3f-42dd-ba8f-017131416961\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"
Oct 30 18:33:21.548406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.548215    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c80b2cb2-7c3f-42dd-ba8f-017131416961-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kk6bc\" (UID: \"c80b2cb2-7c3f-42dd-ba8f-017131416961\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"
Oct 30 18:33:21.548406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.548249    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c80b2cb2-7c3f-42dd-ba8f-017131416961-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kk6bc\" (UID: \"c80b2cb2-7c3f-42dd-ba8f-017131416961\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"
Oct 30 18:33:21.548406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.548288    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-oauth-config\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.548406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.548314    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-trusted-ca-bundle\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.548406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.548342    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-service-ca\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.548406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.548369    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-config\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.548406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.548407    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c80b2cb2-7c3f-42dd-ba8f-017131416961-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kk6bc\" (UID: \"c80b2cb2-7c3f-42dd-ba8f-017131416961\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"
Oct 30 18:33:21.548781 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.548438    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-serving-cert\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.548781 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.548469    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-tmp\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.548781 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.548528    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-oauth-serving-cert\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.548781 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.548557    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t88k6\" (UniqueName: \"kubernetes.io/projected/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-kube-api-access-t88k6\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.549848 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.549702    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c80b2cb2-7c3f-42dd-ba8f-017131416961-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kk6bc\" (UID: \"c80b2cb2-7c3f-42dd-ba8f-017131416961\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"
Oct 30 18:33:21.558962 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.558910    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c80b2cb2-7c3f-42dd-ba8f-017131416961-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kk6bc\" (UID: \"c80b2cb2-7c3f-42dd-ba8f-017131416961\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"
Oct 30 18:33:21.579754 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.579702    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5trmt\" (UniqueName: \"kubernetes.io/projected/c80b2cb2-7c3f-42dd-ba8f-017131416961-kube-api-access-5trmt\") pod \"prometheus-operator-9cd89b74c-kk6bc\" (UID: \"c80b2cb2-7c3f-42dd-ba8f-017131416961\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"
Oct 30 18:33:21.586309 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.586212    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c80b2cb2-7c3f-42dd-ba8f-017131416961-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kk6bc\" (UID: \"c80b2cb2-7c3f-42dd-ba8f-017131416961\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"
Oct 30 18:33:21.649138 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.649087    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-oauth-config\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.649275 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.649195    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-trusted-ca-bundle\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.649275 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.649229    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-service-ca\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.649275 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.649262    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-config\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.649429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.649307    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-serving-cert\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.649429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.649335    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-tmp\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.649429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.649384    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-oauth-serving-cert\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.649429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.649415    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t88k6\" (UniqueName: \"kubernetes.io/projected/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-kube-api-access-t88k6\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.653405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.651462    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-config\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.653405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.652476    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-trusted-ca-bundle\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.653405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.652849    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-tmp\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.653405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.653000    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-service-ca\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.653405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.653358    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-oauth-serving-cert\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.654847 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.654808    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-oauth-config\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.667628 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.667582    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-serving-cert\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.668936 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.668895    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t88k6\" (UniqueName: \"kubernetes.io/projected/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-kube-api-access-t88k6\") pod \"console-8b88fb85b-6bwlm\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") " pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:21.718114 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.718055    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"
Oct 30 18:33:21.757313 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:21.757243    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:22.040158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:22.032723    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc"]
Oct 30 18:33:22.040158 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:22.036672    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80b2cb2_7c3f_42dd_ba8f_017131416961.slice/crio-cf7f1cbef440456fee246e4446bc5d04d7934ed478f9e4203bf3e3989cf85403 WatchSource:0}: Error finding container cf7f1cbef440456fee246e4446bc5d04d7934ed478f9e4203bf3e3989cf85403: Status 404 returned error can't find the container with id cf7f1cbef440456fee246e4446bc5d04d7934ed478f9e4203bf3e3989cf85403
Oct 30 18:33:22.089019 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:22.088907    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b88fb85b-6bwlm"]
Oct 30 18:33:22.097937 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:22.097889    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea63c9be_4d4c_4122_92b9_3eb72cd763a3.slice/crio-d231bd0b548b7f30151ab68879528dfa4842ef0d3e066a59a4ace9c1e1e0967c WatchSource:0}: Error finding container d231bd0b548b7f30151ab68879528dfa4842ef0d3e066a59a4ace9c1e1e0967c: Status 404 returned error can't find the container with id d231bd0b548b7f30151ab68879528dfa4842ef0d3e066a59a4ace9c1e1e0967c
Oct 30 18:33:22.393883 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:22.393838    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2j794" event={"ID":"74d61d4b-1a8f-4256-aae6-c51763689c92","Type":"ContainerStarted","Data":"0ab350d0cb3240902754403f8c6ed7972444452e67d1adf7a2efe07785d8ac1c"}
Oct 30 18:33:22.394413 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:22.393925    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2j794" event={"ID":"74d61d4b-1a8f-4256-aae6-c51763689c92","Type":"ContainerStarted","Data":"bd0fa26210c3fa49c22f32fb421bdc2a48b7ebf1169bc1f8fc58e3f5a734fd0a"}
Oct 30 18:33:22.394413 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:22.394392    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:22.400069 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:22.400030    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc" event={"ID":"c80b2cb2-7c3f-42dd-ba8f-017131416961","Type":"ContainerStarted","Data":"cf7f1cbef440456fee246e4446bc5d04d7934ed478f9e4203bf3e3989cf85403"}
Oct 30 18:33:22.405400 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:22.405302    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b88fb85b-6bwlm" event={"ID":"ea63c9be-4d4c-4122-92b9-3eb72cd763a3","Type":"ContainerStarted","Data":"d231bd0b548b7f30151ab68879528dfa4842ef0d3e066a59a4ace9c1e1e0967c"}
Oct 30 18:33:22.417470 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:22.417385    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-2j794" podStartSLOduration=1.930499765 podStartE2EDuration="5.417358929s" totalImagesPullingTime="3.486859164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:22.416599893 +0000 UTC m=+16.285018227" watchObservedRunningTime="2026-10-30 18:33:22.417358929 +0000 UTC m=+16.285777258"
Oct 30 18:33:23.416040 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:23.415994    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2t9nh" event={"ID":"76637ad2-0d7d-4c03-a808-13e0125424a2","Type":"ContainerStarted","Data":"07cc642916ca12b5ba6fd94d350fd0dcea76d6218ce51ba09fd56a5140149c47"}
Oct 30 18:33:23.419507 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:23.419468    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xn6pz-hnbfq" event={"ID":"c57a61f9-b181-4622-8e40-68402dde03fc","Type":"ContainerStarted","Data":"731cf48d74f3088b6a37499145b4677c4fff5799ccd38274a1b080f11b2e4924"}
Oct 30 18:33:23.471118 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:23.469093    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2t9nh" podStartSLOduration=2.460854669 podStartE2EDuration="6.469061861s" totalImagesPullingTime="4.008207192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:23.445082251 +0000 UTC m=+17.313500582" watchObservedRunningTime="2026-10-30 18:33:23.469061861 +0000 UTC m=+17.337480192"
Oct 30 18:33:24.426763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:24.426704    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-89d5c458-w78vx" event={"ID":"795ba6e5-0a07-487c-b349-772ac223ffab","Type":"ContainerStarted","Data":"c45cad661ab18e73214a28546d8ca73d209c14ea29193c218a5a2869e8e341ef"}
Oct 30 18:33:24.429584 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:24.429538    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vgjw8" event={"ID":"dae5359a-4e6c-4a5d-a688-aff0f02cb240","Type":"ContainerStarted","Data":"d6253899441b4bdd207f23302f979e711e50b81b34fa455c3a8aad9ae9b2547f"}
Oct 30 18:33:24.433846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:24.433710    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc" event={"ID":"c80b2cb2-7c3f-42dd-ba8f-017131416961","Type":"ContainerStarted","Data":"f611b079e18f2926a2ea26a680653762bd2ca09f300fa97762a021e03b855e92"}
Oct 30 18:33:24.433846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:24.433784    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc" event={"ID":"c80b2cb2-7c3f-42dd-ba8f-017131416961","Type":"ContainerStarted","Data":"b59ccfc7c527d8c5071f0694d33b290ea48ed577fde1a67299409f3442f53903"}
Oct 30 18:33:24.453185 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:24.448321    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-xn6pz-hnbfq" podStartSLOduration=3.079588073 podStartE2EDuration="7.448293769s" totalImagesPullingTime="4.368705696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:23.471264869 +0000 UTC m=+17.339683195" watchObservedRunningTime="2026-10-30 18:33:24.448293769 +0000 UTC m=+18.316712099"
Oct 30 18:33:24.453185 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:24.449036    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-89d5c458-w78vx" podStartSLOduration=1.225483257 podStartE2EDuration="7.449018613s" totalImagesPullingTime="6.223535356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:24.447719217 +0000 UTC m=+18.316137548" watchObservedRunningTime="2026-10-30 18:33:24.449018613 +0000 UTC m=+18.317436943"
Oct 30 18:33:24.469521 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:24.469430    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vgjw8" podStartSLOduration=1.723362658 podStartE2EDuration="7.469403964s" totalImagesPullingTime="5.746041306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:24.466936733 +0000 UTC m=+18.335355064" watchObservedRunningTime="2026-10-30 18:33:24.469403964 +0000 UTC m=+18.337822297"
Oct 30 18:33:24.494445 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:24.494343    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kk6bc" podStartSLOduration=1.425654306 podStartE2EDuration="3.494312847s" totalImagesPullingTime="2.068658541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:24.493018121 +0000 UTC m=+18.361436453" watchObservedRunningTime="2026-10-30 18:33:24.494312847 +0000 UTC m=+18.362731178"
Oct 30 18:33:25.435229 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:25.435194    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-89d5c458-w78vx_795ba6e5-0a07-487c-b349-772ac223ffab/console/0.log"
Oct 30 18:33:25.440458 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:25.440410    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b88fb85b-6bwlm" event={"ID":"ea63c9be-4d4c-4122-92b9-3eb72cd763a3","Type":"ContainerStarted","Data":"08459c03c882c6de25fda0cec7f4c3a17682954be21cce2f0407fe675915fca5"}
Oct 30 18:33:25.463642 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:25.463438    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8b88fb85b-6bwlm" podStartSLOduration=1.712915063 podStartE2EDuration="4.463417225s" totalImagesPullingTime="2.750502162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:25.462938565 +0000 UTC m=+19.331356919" watchObservedRunningTime="2026-10-30 18:33:25.463417225 +0000 UTC m=+19.331835557"
Oct 30 18:33:25.620749 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:25.620712    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8b88fb85b-6bwlm_ea63c9be-4d4c-4122-92b9-3eb72cd763a3/console/0.log"
Oct 30 18:33:26.003166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:26.003085    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2j794_74d61d4b-1a8f-4256-aae6-c51763689c92/dns/0.log"
Oct 30 18:33:26.203444 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:26.203403    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2j794_74d61d4b-1a8f-4256-aae6-c51763689c92/kube-rbac-proxy/0.log"
Oct 30 18:33:26.902443 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:26.902399    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sbdjx"]
Oct 30 18:33:26.906459 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:26.906424    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:26.976931 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:26.976885    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 18:33:26.977312 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:26.977286    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 18:33:26.977412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:26.977309    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hh248\""
Oct 30 18:33:26.978096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:26.978071    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 18:33:27.002182 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.002150    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h64hp_9c151e21-28d5-4d4a-8b95-b92d623582c5/dns-node-resolver/0.log"
Oct 30 18:33:27.011646 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.011600    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd3ac4bf-016e-496c-9895-c9fd1d60d897-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.011872 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.011716    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bd3ac4bf-016e-496c-9895-c9fd1d60d897-node-exporter-wtmp\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.011872 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.011770    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd3ac4bf-016e-496c-9895-c9fd1d60d897-node-exporter-tls\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.011872 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.011805    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzp52\" (UniqueName: \"kubernetes.io/projected/bd3ac4bf-016e-496c-9895-c9fd1d60d897-kube-api-access-pzp52\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.011986 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.011918    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd3ac4bf-016e-496c-9895-c9fd1d60d897-metrics-client-ca\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.011986 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.011949    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bd3ac4bf-016e-496c-9895-c9fd1d60d897-node-exporter-accelerators-collector-config\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.011986 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.011975    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd3ac4bf-016e-496c-9895-c9fd1d60d897-sys\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.012174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.012150    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bd3ac4bf-016e-496c-9895-c9fd1d60d897-root\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.012234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.012199    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bd3ac4bf-016e-496c-9895-c9fd1d60d897-node-exporter-textfile\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.112961 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.112917    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd3ac4bf-016e-496c-9895-c9fd1d60d897-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.112961 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.112962    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bd3ac4bf-016e-496c-9895-c9fd1d60d897-node-exporter-wtmp\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.113242 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.112988    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd3ac4bf-016e-496c-9895-c9fd1d60d897-node-exporter-tls\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.113242 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.113020    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzp52\" (UniqueName: \"kubernetes.io/projected/bd3ac4bf-016e-496c-9895-c9fd1d60d897-kube-api-access-pzp52\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.113242 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.113058    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd3ac4bf-016e-496c-9895-c9fd1d60d897-metrics-client-ca\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.113242 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.113082    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bd3ac4bf-016e-496c-9895-c9fd1d60d897-node-exporter-accelerators-collector-config\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.113242 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.113109    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd3ac4bf-016e-496c-9895-c9fd1d60d897-sys\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.113242 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.113148    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bd3ac4bf-016e-496c-9895-c9fd1d60d897-node-exporter-wtmp\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.113242 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.113187    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bd3ac4bf-016e-496c-9895-c9fd1d60d897-root\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.113242 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.113231    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bd3ac4bf-016e-496c-9895-c9fd1d60d897-root\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.113242 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.113237    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bd3ac4bf-016e-496c-9895-c9fd1d60d897-node-exporter-textfile\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.113650 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.113515    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd3ac4bf-016e-496c-9895-c9fd1d60d897-sys\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.113650 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.113566    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bd3ac4bf-016e-496c-9895-c9fd1d60d897-node-exporter-textfile\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.113845 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.113819    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bd3ac4bf-016e-496c-9895-c9fd1d60d897-node-exporter-accelerators-collector-config\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.113972 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.113901    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd3ac4bf-016e-496c-9895-c9fd1d60d897-metrics-client-ca\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.116402 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.116366    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd3ac4bf-016e-496c-9895-c9fd1d60d897-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.116608 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.116366    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd3ac4bf-016e-496c-9895-c9fd1d60d897-node-exporter-tls\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.128231 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.128187    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzp52\" (UniqueName: \"kubernetes.io/projected/bd3ac4bf-016e-496c-9895-c9fd1d60d897-kube-api-access-pzp52\") pod \"node-exporter-sbdjx\" (UID: \"bd3ac4bf-016e-496c-9895-c9fd1d60d897\") " pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.228193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.228055    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sbdjx"
Oct 30 18:33:27.245952 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:27.245890    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ac4bf_016e_496c_9895_c9fd1d60d897.slice/crio-536a04e69932d8bc6a56c2dc8eab332f6988478f30c5d7f3467438c7e707461a WatchSource:0}: Error finding container 536a04e69932d8bc6a56c2dc8eab332f6988478f30c5d7f3467438c7e707461a: Status 404 returned error can't find the container with id 536a04e69932d8bc6a56c2dc8eab332f6988478f30c5d7f3467438c7e707461a
Oct 30 18:33:27.450975 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.450928    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbdjx" event={"ID":"bd3ac4bf-016e-496c-9895-c9fd1d60d897","Type":"ContainerStarted","Data":"536a04e69932d8bc6a56c2dc8eab332f6988478f30c5d7f3467438c7e707461a"}
Oct 30 18:33:27.711985 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:33:27.711930    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:33:27.726149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.726086    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:27.727085 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.727044    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:27.732668 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.732632    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:27.803094 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:27.803051    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-fd56c4b66-72fh4_fa15387d-68c0-4a12-a921-313ba8743b0a/registry/0.log"
Oct 30 18:33:28.002294 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:28.002194    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9mqb5_382d4192-0bfc-4626-820d-5cf183296180/node-ca/0.log"
Oct 30 18:33:28.459398 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:28.459257    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbdjx" event={"ID":"bd3ac4bf-016e-496c-9895-c9fd1d60d897","Type":"ContainerStarted","Data":"022d287713975896a2f26db9bb7ba7f54ce08b858fe3e49c51f97ecb93ebc32c"}
Oct 30 18:33:28.464814 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:28.464781    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:29.002784 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:29.002739    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vgjw8_dae5359a-4e6c-4a5d-a688-aff0f02cb240/serve-healthcheck-canary/0.log"
Oct 30 18:33:29.471240 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:29.471179    4372 generic.go:356] "Generic (PLEG): container finished" podID="bd3ac4bf-016e-496c-9895-c9fd1d60d897" containerID="022d287713975896a2f26db9bb7ba7f54ce08b858fe3e49c51f97ecb93ebc32c" exitCode=0
Oct 30 18:33:29.471593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:29.471285    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbdjx" event={"ID":"bd3ac4bf-016e-496c-9895-c9fd1d60d897","Type":"ContainerDied","Data":"022d287713975896a2f26db9bb7ba7f54ce08b858fe3e49c51f97ecb93ebc32c"}
Oct 30 18:33:30.381677 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:30.381631    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pz2f"
Oct 30 18:33:30.479565 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:30.479511    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbdjx" event={"ID":"bd3ac4bf-016e-496c-9895-c9fd1d60d897","Type":"ContainerStarted","Data":"3f80d249520ae178a0a4b3435159d511ed8d16cece26eddf4edc76933efc3e81"}
Oct 30 18:33:30.479822 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:30.479598    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbdjx" event={"ID":"bd3ac4bf-016e-496c-9895-c9fd1d60d897","Type":"ContainerStarted","Data":"bbaf9244c48704145403c92b352d0e8f83e985913680845518c83a552b980f86"}
Oct 30 18:33:30.505305 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:30.505235    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sbdjx" podStartSLOduration=3.349587813 podStartE2EDuration="4.505214892s" totalImagesPullingTime="1.050405925s" totalInitContainerRuntime="105.221154ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:30.503617897 +0000 UTC m=+24.372036232" watchObservedRunningTime="2026-10-30 18:33:30.505214892 +0000 UTC m=+24.373633222"
Oct 30 18:33:31.758591 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:31.758548    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:31.759202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:31.758632    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:31.765018 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:31.764982    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:32.422117 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:32.422083    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2j794"
Oct 30 18:33:32.495364 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:32.495322    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:33:32.556659 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:32.556607    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-89d5c458-w78vx"]
Oct 30 18:33:33.813798 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:33.812189    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-54645595b4-kxm7v"]
Oct 30 18:33:33.820186 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:33.820141    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:33.835452 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:33.835392    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54645595b4-kxm7v"]
Oct 30 18:33:33.979703 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:33.979654    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-console-config\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:33.979974 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:33.979784    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83a07661-3916-4026-9853-4edb1210cf2d-console-serving-cert\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:33.979974 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:33.979829    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a07661-3916-4026-9853-4edb1210cf2d-tmp\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:33.980104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:33.979974    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83a07661-3916-4026-9853-4edb1210cf2d-console-oauth-config\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:33.980104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:33.980027    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-service-ca\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:33.980104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:33.980098    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-trusted-ca-bundle\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:33.980262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:33.980175    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7x8l\" (UniqueName: \"kubernetes.io/projected/83a07661-3916-4026-9853-4edb1210cf2d-kube-api-access-f7x8l\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:33.980307 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:33.980277    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-oauth-serving-cert\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.080824 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.080772    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a07661-3916-4026-9853-4edb1210cf2d-tmp\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.080824 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.080829    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83a07661-3916-4026-9853-4edb1210cf2d-console-oauth-config\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.081193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.080863    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-service-ca\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.081193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.080907    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-trusted-ca-bundle\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.081193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.080966    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7x8l\" (UniqueName: \"kubernetes.io/projected/83a07661-3916-4026-9853-4edb1210cf2d-kube-api-access-f7x8l\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.081193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.081017    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-oauth-serving-cert\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.081193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.081045    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-console-config\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.081193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.081094    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83a07661-3916-4026-9853-4edb1210cf2d-console-serving-cert\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.081437 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.081386    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a07661-3916-4026-9853-4edb1210cf2d-tmp\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.082477 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.082442    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-service-ca\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.083032 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.083005    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-oauth-serving-cert\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.083770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.083729    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-console-config\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.083770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.083753    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-trusted-ca-bundle\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.084769 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.084747    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83a07661-3916-4026-9853-4edb1210cf2d-console-serving-cert\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.084877 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.084847    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83a07661-3916-4026-9853-4edb1210cf2d-console-oauth-config\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.096662 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.096617    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7x8l\" (UniqueName: \"kubernetes.io/projected/83a07661-3916-4026-9853-4edb1210cf2d-kube-api-access-f7x8l\") pod \"console-54645595b4-kxm7v\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") " pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.139508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.139448    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:34.438683 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.438549    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54645595b4-kxm7v"]
Oct 30 18:33:34.443851 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:33:34.443817    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a07661_3916_4026_9853_4edb1210cf2d.slice/crio-3470436a4d8d86ed6f053f498532f67793e82ecae8ec40eb8afeb9dbd64de538 WatchSource:0}: Error finding container 3470436a4d8d86ed6f053f498532f67793e82ecae8ec40eb8afeb9dbd64de538: Status 404 returned error can't find the container with id 3470436a4d8d86ed6f053f498532f67793e82ecae8ec40eb8afeb9dbd64de538
Oct 30 18:33:34.501554 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:34.501459    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54645595b4-kxm7v" event={"ID":"83a07661-3916-4026-9853-4edb1210cf2d","Type":"ContainerStarted","Data":"3470436a4d8d86ed6f053f498532f67793e82ecae8ec40eb8afeb9dbd64de538"}
Oct 30 18:33:35.508366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:35.508318    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54645595b4-kxm7v" event={"ID":"83a07661-3916-4026-9853-4edb1210cf2d","Type":"ContainerStarted","Data":"9ae53ffe1324ed5b738301fb4e7ea6e4562fdaa1c98d5bd2a74cefa5af369513"}
Oct 30 18:33:35.532540 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:35.532470    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-54645595b4-kxm7v" podStartSLOduration=2.5324417070000003 podStartE2EDuration="2.532441707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:35.531387815 +0000 UTC m=+29.399806168" watchObservedRunningTime="2026-10-30 18:33:35.532441707 +0000 UTC m=+29.400860141"
Oct 30 18:33:38.303943 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:38.303904    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n52lm"
Oct 30 18:33:38.527745 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:38.527684    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-fx7q7" event={"ID":"d02d01c9-1b8c-4632-866c-3fecfda75c4a","Type":"ContainerStarted","Data":"572bc5d58566ca1a62c2ba427f34b2389375f8ec9bc41ef90b78368c63933d89"}
Oct 30 18:33:38.528102 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:38.527987    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-fx7q7"
Oct 30 18:33:38.529933 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:38.529907    4372 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-fx7q7 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.7:8080/\": dial tcp 10.133.0.7:8080: connect: connection refused" start-of-body=
Oct 30 18:33:38.530020 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:38.529962    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-fx7q7" podUID="d02d01c9-1b8c-4632-866c-3fecfda75c4a" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.7:8080/\": dial tcp 10.133.0.7:8080: connect: connection refused"
Oct 30 18:33:38.555087 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:38.554946    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-fx7q7" podStartSLOduration=1.253005166 podStartE2EDuration="21.554926265s" totalImagesPullingTime="20.301921099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:38.55440816 +0000 UTC m=+32.422826491" watchObservedRunningTime="2026-10-30 18:33:38.554926265 +0000 UTC m=+32.423344596"
Oct 30 18:33:39.362278 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:39.362239    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fd56c4b66-72fh4"
Oct 30 18:33:39.551988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:39.551949    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-fx7q7"
Oct 30 18:33:44.139972 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:44.139910    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:44.140633 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:44.140021    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:44.146595 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:44.146558    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:44.576764 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:44.576731    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:33:44.645397 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:44.645341    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8b88fb85b-6bwlm"]
Oct 30 18:33:57.583826 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.583679    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-89d5c458-w78vx" podUID="795ba6e5-0a07-487c-b349-772ac223ffab" containerName="console" containerID="cri-o://c45cad661ab18e73214a28546d8ca73d209c14ea29193c218a5a2869e8e341ef" gracePeriod=15
Oct 30 18:33:57.701627 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:33:57.701582    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:33:57.868572 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.868534    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:57.916085 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.916039    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/795ba6e5-0a07-487c-b349-772ac223ffab-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/795ba6e5-0a07-487c-b349-772ac223ffab-console-oauth-config\") pod \"795ba6e5-0a07-487c-b349-772ac223ffab\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") "
Oct 30 18:33:57.916336 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.916100    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-oauth-serving-cert\") pod \"795ba6e5-0a07-487c-b349-772ac223ffab\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") "
Oct 30 18:33:57.916336 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.916144    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/795ba6e5-0a07-487c-b349-772ac223ffab-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/795ba6e5-0a07-487c-b349-772ac223ffab-console-serving-cert\") pod \"795ba6e5-0a07-487c-b349-772ac223ffab\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") "
Oct 30 18:33:57.916336 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.916301    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-service-ca\" (UniqueName: \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-service-ca\") pod \"795ba6e5-0a07-487c-b349-772ac223ffab\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") "
Oct 30 18:33:57.916445 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.916360    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-console-config\" (UniqueName: \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-console-config\") pod \"795ba6e5-0a07-487c-b349-772ac223ffab\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") "
Oct 30 18:33:57.916445 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.916400    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795ba6e5-0a07-487c-b349-772ac223ffab-kube-api-access-sx8fg\" (UniqueName: \"kubernetes.io/projected/795ba6e5-0a07-487c-b349-772ac223ffab-kube-api-access-sx8fg\") pod \"795ba6e5-0a07-487c-b349-772ac223ffab\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") "
Oct 30 18:33:57.916540 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.916451    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795ba6e5-0a07-487c-b349-772ac223ffab-tmp\" (UniqueName: \"kubernetes.io/empty-dir/795ba6e5-0a07-487c-b349-772ac223ffab-tmp\") pod \"795ba6e5-0a07-487c-b349-772ac223ffab\" (UID: \"795ba6e5-0a07-487c-b349-772ac223ffab\") "
Oct 30 18:33:57.916712 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.916637    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-oauth-serving-cert" pod "795ba6e5-0a07-487c-b349-772ac223ffab" (UID: "795ba6e5-0a07-487c-b349-772ac223ffab"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:33:57.916712 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.916682    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-service-ca" pod "795ba6e5-0a07-487c-b349-772ac223ffab" (UID: "795ba6e5-0a07-487c-b349-772ac223ffab"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:33:57.916897 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.916757    4372 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-oauth-serving-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:57.916897 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.916762    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-console-config" pod "795ba6e5-0a07-487c-b349-772ac223ffab" (UID: "795ba6e5-0a07-487c-b349-772ac223ffab"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:33:57.917081 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.916900    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795ba6e5-0a07-487c-b349-772ac223ffab-tmp" pod "795ba6e5-0a07-487c-b349-772ac223ffab" (UID: "795ba6e5-0a07-487c-b349-772ac223ffab"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:33:57.919172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.919110    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795ba6e5-0a07-487c-b349-772ac223ffab-console-serving-cert" pod "795ba6e5-0a07-487c-b349-772ac223ffab" (UID: "795ba6e5-0a07-487c-b349-772ac223ffab"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:33:57.919404 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.919378    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795ba6e5-0a07-487c-b349-772ac223ffab-console-oauth-config" pod "795ba6e5-0a07-487c-b349-772ac223ffab" (UID: "795ba6e5-0a07-487c-b349-772ac223ffab"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:33:57.919404 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:57.919387    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795ba6e5-0a07-487c-b349-772ac223ffab-kube-api-access-sx8fg" pod "795ba6e5-0a07-487c-b349-772ac223ffab" (UID: "795ba6e5-0a07-487c-b349-772ac223ffab"). InnerVolumeSpecName "kube-api-access-sx8fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:33:58.017444 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:58.017395    4372 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/795ba6e5-0a07-487c-b349-772ac223ffab-console-oauth-config\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:58.017444 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:58.017439    4372 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/795ba6e5-0a07-487c-b349-772ac223ffab-console-serving-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:58.017444 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:58.017450    4372 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:58.017444 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:58.017460    4372 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/795ba6e5-0a07-487c-b349-772ac223ffab-console-config\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:58.017781 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:58.017470    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx8fg\" (UniqueName: \"kubernetes.io/projected/795ba6e5-0a07-487c-b349-772ac223ffab-kube-api-access-sx8fg\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:58.017781 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:58.017483    4372 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/795ba6e5-0a07-487c-b349-772ac223ffab-tmp\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:33:58.634558 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:58.634504    4372 generic.go:356] "Generic (PLEG): container finished" podID="795ba6e5-0a07-487c-b349-772ac223ffab" containerID="c45cad661ab18e73214a28546d8ca73d209c14ea29193c218a5a2869e8e341ef" exitCode=0
Oct 30 18:33:58.635078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:58.634572    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-89d5c458-w78vx"
Oct 30 18:33:58.635078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:58.634596    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-89d5c458-w78vx" event={"ID":"795ba6e5-0a07-487c-b349-772ac223ffab","Type":"ContainerDied","Data":"c45cad661ab18e73214a28546d8ca73d209c14ea29193c218a5a2869e8e341ef"}
Oct 30 18:33:58.635078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:58.634667    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-89d5c458-w78vx" event={"ID":"795ba6e5-0a07-487c-b349-772ac223ffab","Type":"ContainerDied","Data":"d168081812e8d5f830c1f95cdad76b1affb455795c5d8e50fa0bab2b0374cf8a"}
Oct 30 18:33:58.635078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:58.634704    4372 scope.go:122] "RemoveContainer" containerID="c45cad661ab18e73214a28546d8ca73d209c14ea29193c218a5a2869e8e341ef"
Oct 30 18:33:58.644807 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:58.644776    4372 scope.go:122] "RemoveContainer" containerID="c45cad661ab18e73214a28546d8ca73d209c14ea29193c218a5a2869e8e341ef"
Oct 30 18:33:58.645242 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:33:58.645216    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c45cad661ab18e73214a28546d8ca73d209c14ea29193c218a5a2869e8e341ef\": container with ID starting with c45cad661ab18e73214a28546d8ca73d209c14ea29193c218a5a2869e8e341ef not found: ID does not exist" containerID="c45cad661ab18e73214a28546d8ca73d209c14ea29193c218a5a2869e8e341ef"
Oct 30 18:33:58.645315 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:58.645270    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45cad661ab18e73214a28546d8ca73d209c14ea29193c218a5a2869e8e341ef"} err="failed to get container status \"c45cad661ab18e73214a28546d8ca73d209c14ea29193c218a5a2869e8e341ef\": rpc error: code = NotFound desc = could not find container \"c45cad661ab18e73214a28546d8ca73d209c14ea29193c218a5a2869e8e341ef\": container with ID starting with c45cad661ab18e73214a28546d8ca73d209c14ea29193c218a5a2869e8e341ef not found: ID does not exist"
Oct 30 18:33:58.655059 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:58.655022    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-89d5c458-w78vx"]
Oct 30 18:33:58.659300 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:33:58.659236    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-89d5c458-w78vx"]
Oct 30 18:34:00.263607 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:00.263563    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="795ba6e5-0a07-487c-b349-772ac223ffab" path="/var/lib/kubelet/pods/795ba6e5-0a07-487c-b349-772ac223ffab/volumes"
Oct 30 18:34:09.672410 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:09.672336    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8b88fb85b-6bwlm" podUID="ea63c9be-4d4c-4122-92b9-3eb72cd763a3" containerName="console" containerID="cri-o://08459c03c882c6de25fda0cec7f4c3a17682954be21cce2f0407fe675915fca5" gracePeriod=15
Oct 30 18:34:09.972987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:09.972932    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:34:10.023287 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.023246    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-config\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-config\") pod \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") "
Oct 30 18:34:10.023533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.023316    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-serving-cert\") pod \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") "
Oct 30 18:34:10.023533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.023339    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-trusted-ca-bundle\") pod \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") "
Oct 30 18:34:10.023533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.023370    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-kube-api-access-t88k6\" (UniqueName: \"kubernetes.io/projected/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-kube-api-access-t88k6\") pod \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") "
Oct 30 18:34:10.023533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.023401    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-oauth-config\") pod \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") "
Oct 30 18:34:10.023533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.023449    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-oauth-serving-cert\") pod \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") "
Oct 30 18:34:10.023533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.023479    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-tmp\") pod \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") "
Oct 30 18:34:10.023533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.023512    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-service-ca\") pod \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\" (UID: \"ea63c9be-4d4c-4122-92b9-3eb72cd763a3\") "
Oct 30 18:34:10.023914 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.023745    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-config" pod "ea63c9be-4d4c-4122-92b9-3eb72cd763a3" (UID: "ea63c9be-4d4c-4122-92b9-3eb72cd763a3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:34:10.024002 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.023979    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-oauth-serving-cert" pod "ea63c9be-4d4c-4122-92b9-3eb72cd763a3" (UID: "ea63c9be-4d4c-4122-92b9-3eb72cd763a3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:34:10.024002 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.023987    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-trusted-ca-bundle" pod "ea63c9be-4d4c-4122-92b9-3eb72cd763a3" (UID: "ea63c9be-4d4c-4122-92b9-3eb72cd763a3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:34:10.024149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.024105    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-service-ca" pod "ea63c9be-4d4c-4122-92b9-3eb72cd763a3" (UID: "ea63c9be-4d4c-4122-92b9-3eb72cd763a3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:34:10.024203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.024104    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-tmp" pod "ea63c9be-4d4c-4122-92b9-3eb72cd763a3" (UID: "ea63c9be-4d4c-4122-92b9-3eb72cd763a3"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:34:10.026096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.026064    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-oauth-config" pod "ea63c9be-4d4c-4122-92b9-3eb72cd763a3" (UID: "ea63c9be-4d4c-4122-92b9-3eb72cd763a3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:34:10.026096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.026080    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-kube-api-access-t88k6" pod "ea63c9be-4d4c-4122-92b9-3eb72cd763a3" (UID: "ea63c9be-4d4c-4122-92b9-3eb72cd763a3"). InnerVolumeSpecName "kube-api-access-t88k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:34:10.026096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.026084    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-serving-cert" pod "ea63c9be-4d4c-4122-92b9-3eb72cd763a3" (UID: "ea63c9be-4d4c-4122-92b9-3eb72cd763a3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:34:10.124111 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.124065    4372 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-config\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:10.124111 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.124105    4372 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-serving-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:10.124368 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.124115    4372 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-trusted-ca-bundle\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:10.124368 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.124156    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t88k6\" (UniqueName: \"kubernetes.io/projected/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-kube-api-access-t88k6\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:10.124368 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.124167    4372 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-console-oauth-config\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:10.124368 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.124176    4372 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-oauth-serving-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:10.124368 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.124185    4372 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-tmp\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:10.124368 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.124193    4372 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea63c9be-4d4c-4122-92b9-3eb72cd763a3-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:10.683717 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.683673    4372 generic.go:356] "Generic (PLEG): container finished" podID="ea63c9be-4d4c-4122-92b9-3eb72cd763a3" containerID="08459c03c882c6de25fda0cec7f4c3a17682954be21cce2f0407fe675915fca5" exitCode=0
Oct 30 18:34:10.684308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.683738    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b88fb85b-6bwlm"
Oct 30 18:34:10.684308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.683752    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b88fb85b-6bwlm" event={"ID":"ea63c9be-4d4c-4122-92b9-3eb72cd763a3","Type":"ContainerDied","Data":"08459c03c882c6de25fda0cec7f4c3a17682954be21cce2f0407fe675915fca5"}
Oct 30 18:34:10.684308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.683800    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b88fb85b-6bwlm" event={"ID":"ea63c9be-4d4c-4122-92b9-3eb72cd763a3","Type":"ContainerDied","Data":"d231bd0b548b7f30151ab68879528dfa4842ef0d3e066a59a4ace9c1e1e0967c"}
Oct 30 18:34:10.684308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.683825    4372 scope.go:122] "RemoveContainer" containerID="08459c03c882c6de25fda0cec7f4c3a17682954be21cce2f0407fe675915fca5"
Oct 30 18:34:10.694026 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.693993    4372 scope.go:122] "RemoveContainer" containerID="08459c03c882c6de25fda0cec7f4c3a17682954be21cce2f0407fe675915fca5"
Oct 30 18:34:10.694537 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:34:10.694458    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08459c03c882c6de25fda0cec7f4c3a17682954be21cce2f0407fe675915fca5\": container with ID starting with 08459c03c882c6de25fda0cec7f4c3a17682954be21cce2f0407fe675915fca5 not found: ID does not exist" containerID="08459c03c882c6de25fda0cec7f4c3a17682954be21cce2f0407fe675915fca5"
Oct 30 18:34:10.694537 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.694497    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08459c03c882c6de25fda0cec7f4c3a17682954be21cce2f0407fe675915fca5"} err="failed to get container status \"08459c03c882c6de25fda0cec7f4c3a17682954be21cce2f0407fe675915fca5\": rpc error: code = NotFound desc = could not find container \"08459c03c882c6de25fda0cec7f4c3a17682954be21cce2f0407fe675915fca5\": container with ID starting with 08459c03c882c6de25fda0cec7f4c3a17682954be21cce2f0407fe675915fca5 not found: ID does not exist"
Oct 30 18:34:10.704400 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.704347    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8b88fb85b-6bwlm"]
Oct 30 18:34:10.714612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:10.714569    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8b88fb85b-6bwlm"]
Oct 30 18:34:12.262588 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:12.262530    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea63c9be-4d4c-4122-92b9-3eb72cd763a3" path="/var/lib/kubelet/pods/ea63c9be-4d4c-4122-92b9-3eb72cd763a3/volumes"
Oct 30 18:34:18.897047 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:18.897007    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cd4c4444f-42tgx"]
Oct 30 18:34:18.897529 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:18.897406    4372 cpu_manager.go:414] "containerMap: removing container" podUID="ea63c9be-4d4c-4122-92b9-3eb72cd763a3" containerName="console"
Oct 30 18:34:18.897529 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:18.897423    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea63c9be-4d4c-4122-92b9-3eb72cd763a3" containerName="console"
Oct 30 18:34:18.897529 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:18.897436    4372 cpu_manager.go:414] "containerMap: removing container" podUID="795ba6e5-0a07-487c-b349-772ac223ffab" containerName="console"
Oct 30 18:34:18.897529 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:18.897441    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795ba6e5-0a07-487c-b349-772ac223ffab" containerName="console"
Oct 30 18:34:18.897529 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:18.897484    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea63c9be-4d4c-4122-92b9-3eb72cd763a3" containerName="console"
Oct 30 18:34:18.897529 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:18.897492    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="795ba6e5-0a07-487c-b349-772ac223ffab" containerName="console"
Oct 30 18:34:18.920136 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:18.920072    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd4c4444f-42tgx"]
Oct 30 18:34:18.920552 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:18.920518    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.005364 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.005307    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-trusted-ca-bundle\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.005364 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.005366    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb75r\" (UniqueName: \"kubernetes.io/projected/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-kube-api-access-rb75r\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.005621 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.005457    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-serving-cert\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.005621 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.005499    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-service-ca\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.005621 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.005568    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-config\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.005621 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.005603    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-oauth-config\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.005762 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.005658    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-oauth-serving-cert\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.005762 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.005677    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-tmp\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.107035 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.106974    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-trusted-ca-bundle\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.107035 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.107042    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb75r\" (UniqueName: \"kubernetes.io/projected/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-kube-api-access-rb75r\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.107318 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.107062    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-serving-cert\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.107318 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.107087    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-service-ca\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.107318 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.107104    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-config\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.107318 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.107159    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-oauth-config\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.107318 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.107206    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-oauth-serving-cert\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.107318 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.107232    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-tmp\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.107869 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.107829    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-tmp\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.108148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.108097    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-config\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.108253 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.108091    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-service-ca\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.108253 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.108181    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-trusted-ca-bundle\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.108359 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.108298    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-oauth-serving-cert\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.110685 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.110654    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-oauth-config\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.110885 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.110858    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-serving-cert\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.121761 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.121704    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb75r\" (UniqueName: \"kubernetes.io/projected/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-kube-api-access-rb75r\") pod \"console-6cd4c4444f-42tgx\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") " pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.246932 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.246830    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:19.567690 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.567581    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd4c4444f-42tgx"]
Oct 30 18:34:19.571914 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:34:19.571875    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f5ed70_3252_4f2a_8c4b_cb9f7faff5f8.slice/crio-5e38ea78f68f196a1dacc68d51c1f5ed5e4b0453a141c3d4384d53ab4776618f WatchSource:0}: Error finding container 5e38ea78f68f196a1dacc68d51c1f5ed5e4b0453a141c3d4384d53ab4776618f: Status 404 returned error can't find the container with id 5e38ea78f68f196a1dacc68d51c1f5ed5e4b0453a141c3d4384d53ab4776618f
Oct 30 18:34:19.721147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.721081    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd4c4444f-42tgx" event={"ID":"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8","Type":"ContainerStarted","Data":"05540f20452cdeda25d0d883d8575eebf29ca3416052344349bd4bcffc958e58"}
Oct 30 18:34:19.721421 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.721170    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd4c4444f-42tgx" event={"ID":"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8","Type":"ContainerStarted","Data":"5e38ea78f68f196a1dacc68d51c1f5ed5e4b0453a141c3d4384d53ab4776618f"}
Oct 30 18:34:19.745173 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:19.745062    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6cd4c4444f-42tgx" podStartSLOduration=1.7450393320000002 podStartE2EDuration="1.745039332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:19.744105845 +0000 UTC m=+73.612524176" watchObservedRunningTime="2026-10-30 18:34:19.745039332 +0000 UTC m=+73.613457663"
Oct 30 18:34:27.698729 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:34:27.698682    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:34:29.247433 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:29.247376    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:29.247917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:29.247471    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:29.252650 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:29.252610    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:29.769797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:29.769762    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:34:29.821952 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:29.821904    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54645595b4-kxm7v"]
Oct 30 18:34:50.842378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:50.842320    4372 generic.go:356] "Generic (PLEG): container finished" podID="c57a61f9-b181-4622-8e40-68402dde03fc" containerID="731cf48d74f3088b6a37499145b4677c4fff5799ccd38274a1b080f11b2e4924" exitCode=0
Oct 30 18:34:50.842967 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:50.842403    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xn6pz-hnbfq" event={"ID":"c57a61f9-b181-4622-8e40-68402dde03fc","Type":"ContainerDied","Data":"731cf48d74f3088b6a37499145b4677c4fff5799ccd38274a1b080f11b2e4924"}
Oct 30 18:34:51.993889 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:51.993851    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:34:52.087440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:52.087386    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57a61f9-b181-4622-8e40-68402dde03fc-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57a61f9-b181-4622-8e40-68402dde03fc-trusted-ca-bundle\") pod \"c57a61f9-b181-4622-8e40-68402dde03fc\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") "
Oct 30 18:34:52.087440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:52.087435    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57a61f9-b181-4622-8e40-68402dde03fc-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c57a61f9-b181-4622-8e40-68402dde03fc-archives-path\") pod \"c57a61f9-b181-4622-8e40-68402dde03fc\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") "
Oct 30 18:34:52.087750 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:52.087460    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57a61f9-b181-4622-8e40-68402dde03fc-kube-api-access-bmtss\" (UniqueName: \"kubernetes.io/projected/c57a61f9-b181-4622-8e40-68402dde03fc-kube-api-access-bmtss\") pod \"c57a61f9-b181-4622-8e40-68402dde03fc\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") "
Oct 30 18:34:52.087750 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:52.087508    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57a61f9-b181-4622-8e40-68402dde03fc-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57a61f9-b181-4622-8e40-68402dde03fc-service-ca-bundle\") pod \"c57a61f9-b181-4622-8e40-68402dde03fc\" (UID: \"c57a61f9-b181-4622-8e40-68402dde03fc\") "
Oct 30 18:34:52.087858 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:52.087817    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57a61f9-b181-4622-8e40-68402dde03fc-archives-path" pod "c57a61f9-b181-4622-8e40-68402dde03fc" (UID: "c57a61f9-b181-4622-8e40-68402dde03fc"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:34:52.087858 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:52.087840    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57a61f9-b181-4622-8e40-68402dde03fc-trusted-ca-bundle" pod "c57a61f9-b181-4622-8e40-68402dde03fc" (UID: "c57a61f9-b181-4622-8e40-68402dde03fc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:34:52.087970 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:52.087863    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57a61f9-b181-4622-8e40-68402dde03fc-service-ca-bundle" pod "c57a61f9-b181-4622-8e40-68402dde03fc" (UID: "c57a61f9-b181-4622-8e40-68402dde03fc"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:34:52.090371 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:52.090329    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57a61f9-b181-4622-8e40-68402dde03fc-kube-api-access-bmtss" pod "c57a61f9-b181-4622-8e40-68402dde03fc" (UID: "c57a61f9-b181-4622-8e40-68402dde03fc"). InnerVolumeSpecName "kube-api-access-bmtss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:34:52.188086 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:52.187926    4372 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57a61f9-b181-4622-8e40-68402dde03fc-trusted-ca-bundle\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:52.188086 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:52.187986    4372 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c57a61f9-b181-4622-8e40-68402dde03fc-archives-path\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:52.188086 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:52.187998    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmtss\" (UniqueName: \"kubernetes.io/projected/c57a61f9-b181-4622-8e40-68402dde03fc-kube-api-access-bmtss\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:52.188086 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:52.188013    4372 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57a61f9-b181-4622-8e40-68402dde03fc-service-ca-bundle\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:52.855434 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:52.855392    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xn6pz-hnbfq" event={"ID":"c57a61f9-b181-4622-8e40-68402dde03fc","Type":"ContainerDied","Data":"dbd9ce47886a33edff0ea00ff25c6762240094b634c005f35492dc4b5f46a497"}
Oct 30 18:34:52.855434 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:52.855429    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xn6pz-hnbfq"
Oct 30 18:34:52.855676 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:52.855476    4372 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbd9ce47886a33edff0ea00ff25c6762240094b634c005f35492dc4b5f46a497"
Oct 30 18:34:54.844181 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:54.844109    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-54645595b4-kxm7v" podUID="83a07661-3916-4026-9853-4edb1210cf2d" containerName="console" containerID="cri-o://9ae53ffe1324ed5b738301fb4e7ea6e4562fdaa1c98d5bd2a74cefa5af369513" gracePeriod=15
Oct 30 18:34:55.100007 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.099902    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:34:55.109631 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.109587    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-console-config\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-console-config\") pod \"83a07661-3916-4026-9853-4edb1210cf2d\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") "
Oct 30 18:34:55.109631 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.109633    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-trusted-ca-bundle\") pod \"83a07661-3916-4026-9853-4edb1210cf2d\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") "
Oct 30 18:34:55.109861 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.109671    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83a07661-3916-4026-9853-4edb1210cf2d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83a07661-3916-4026-9853-4edb1210cf2d-console-oauth-config\") pod \"83a07661-3916-4026-9853-4edb1210cf2d\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") "
Oct 30 18:34:55.109861 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.109694    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a07661-3916-4026-9853-4edb1210cf2d-kube-api-access-f7x8l\" (UniqueName: \"kubernetes.io/projected/83a07661-3916-4026-9853-4edb1210cf2d-kube-api-access-f7x8l\") pod \"83a07661-3916-4026-9853-4edb1210cf2d\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") "
Oct 30 18:34:55.109861 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.109723    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-service-ca\") pod \"83a07661-3916-4026-9853-4edb1210cf2d\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") "
Oct 30 18:34:55.109861 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.109784    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a07661-3916-4026-9853-4edb1210cf2d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a07661-3916-4026-9853-4edb1210cf2d-tmp\") pod \"83a07661-3916-4026-9853-4edb1210cf2d\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") "
Oct 30 18:34:55.109861 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.109843    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83a07661-3916-4026-9853-4edb1210cf2d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83a07661-3916-4026-9853-4edb1210cf2d-console-serving-cert\") pod \"83a07661-3916-4026-9853-4edb1210cf2d\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") "
Oct 30 18:34:55.110087 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.109877    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-oauth-serving-cert\") pod \"83a07661-3916-4026-9853-4edb1210cf2d\" (UID: \"83a07661-3916-4026-9853-4edb1210cf2d\") "
Oct 30 18:34:55.110191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.110162    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-console-config" pod "83a07661-3916-4026-9853-4edb1210cf2d" (UID: "83a07661-3916-4026-9853-4edb1210cf2d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:34:55.110438 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.110407    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-trusted-ca-bundle" pod "83a07661-3916-4026-9853-4edb1210cf2d" (UID: "83a07661-3916-4026-9853-4edb1210cf2d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:34:55.110560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.110491    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-service-ca" pod "83a07661-3916-4026-9853-4edb1210cf2d" (UID: "83a07661-3916-4026-9853-4edb1210cf2d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:34:55.110560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.110500    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-oauth-serving-cert" pod "83a07661-3916-4026-9853-4edb1210cf2d" (UID: "83a07661-3916-4026-9853-4edb1210cf2d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:34:55.110763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.110576    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a07661-3916-4026-9853-4edb1210cf2d-tmp" pod "83a07661-3916-4026-9853-4edb1210cf2d" (UID: "83a07661-3916-4026-9853-4edb1210cf2d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:34:55.112667 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.112622    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a07661-3916-4026-9853-4edb1210cf2d-kube-api-access-f7x8l" pod "83a07661-3916-4026-9853-4edb1210cf2d" (UID: "83a07661-3916-4026-9853-4edb1210cf2d"). InnerVolumeSpecName "kube-api-access-f7x8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:34:55.112667 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.112632    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a07661-3916-4026-9853-4edb1210cf2d-console-oauth-config" pod "83a07661-3916-4026-9853-4edb1210cf2d" (UID: "83a07661-3916-4026-9853-4edb1210cf2d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:34:55.112875 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.112702    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a07661-3916-4026-9853-4edb1210cf2d-console-serving-cert" pod "83a07661-3916-4026-9853-4edb1210cf2d" (UID: "83a07661-3916-4026-9853-4edb1210cf2d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:34:55.210706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.210658    4372 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a07661-3916-4026-9853-4edb1210cf2d-tmp\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:55.210706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.210698    4372 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83a07661-3916-4026-9853-4edb1210cf2d-console-serving-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:55.210706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.210710    4372 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-oauth-serving-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:55.210706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.210720    4372 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-console-config\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:55.211079 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.210729    4372 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-trusted-ca-bundle\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:55.211079 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.210738    4372 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83a07661-3916-4026-9853-4edb1210cf2d-console-oauth-config\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:55.211079 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.210748    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7x8l\" (UniqueName: \"kubernetes.io/projected/83a07661-3916-4026-9853-4edb1210cf2d-kube-api-access-f7x8l\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:55.211079 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.210758    4372 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83a07661-3916-4026-9853-4edb1210cf2d-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:34:55.213909 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.213874    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-577c7d6796-m7ztx"]
Oct 30 18:34:55.214205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.214191    4372 cpu_manager.go:414] "containerMap: removing container" podUID="c57a61f9-b181-4622-8e40-68402dde03fc" containerName="insights-gathering"
Oct 30 18:34:55.214251 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.214207    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57a61f9-b181-4622-8e40-68402dde03fc" containerName="insights-gathering"
Oct 30 18:34:55.214251 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.214225    4372 cpu_manager.go:414] "containerMap: removing container" podUID="83a07661-3916-4026-9853-4edb1210cf2d" containerName="console"
Oct 30 18:34:55.214251 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.214231    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a07661-3916-4026-9853-4edb1210cf2d" containerName="console"
Oct 30 18:34:55.214343 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.214283    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57a61f9-b181-4622-8e40-68402dde03fc" containerName="insights-gathering"
Oct 30 18:34:55.214343 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.214292    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a07661-3916-4026-9853-4edb1210cf2d" containerName="console"
Oct 30 18:34:55.217525 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.217503    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.229523 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.229473    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-577c7d6796-m7ztx"]
Oct 30 18:34:55.312099 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.312040    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f93f0a1-d934-45f5-9397-98c057e4f050-service-ca\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.312099 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.312092    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f93f0a1-d934-45f5-9397-98c057e4f050-console-serving-cert\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.312099 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.312118    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f93f0a1-d934-45f5-9397-98c057e4f050-tmp\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.312461 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.312223    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f93f0a1-d934-45f5-9397-98c057e4f050-console-oauth-config\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.312461 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.312255    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f93f0a1-d934-45f5-9397-98c057e4f050-oauth-serving-cert\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.312461 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.312363    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f93f0a1-d934-45f5-9397-98c057e4f050-trusted-ca-bundle\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.312461 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.312387    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f93f0a1-d934-45f5-9397-98c057e4f050-console-config\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.312461 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.312425    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl9kc\" (UniqueName: \"kubernetes.io/projected/9f93f0a1-d934-45f5-9397-98c057e4f050-kube-api-access-dl9kc\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.413578 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.413446    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f93f0a1-d934-45f5-9397-98c057e4f050-service-ca\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.413578 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.413497    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f93f0a1-d934-45f5-9397-98c057e4f050-console-serving-cert\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.413578 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.413517    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f93f0a1-d934-45f5-9397-98c057e4f050-tmp\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.413910 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.413678    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f93f0a1-d934-45f5-9397-98c057e4f050-console-oauth-config\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.413910 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.413721    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f93f0a1-d934-45f5-9397-98c057e4f050-oauth-serving-cert\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.413910 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.413770    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f93f0a1-d934-45f5-9397-98c057e4f050-trusted-ca-bundle\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.413910 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.413799    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f93f0a1-d934-45f5-9397-98c057e4f050-console-config\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.413910 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.413866    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl9kc\" (UniqueName: \"kubernetes.io/projected/9f93f0a1-d934-45f5-9397-98c057e4f050-kube-api-access-dl9kc\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.414169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.414058    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f93f0a1-d934-45f5-9397-98c057e4f050-tmp\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.414515 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.414475    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f93f0a1-d934-45f5-9397-98c057e4f050-service-ca\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.414657 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.414628    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f93f0a1-d934-45f5-9397-98c057e4f050-oauth-serving-cert\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.414891 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.414871    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f93f0a1-d934-45f5-9397-98c057e4f050-trusted-ca-bundle\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.415340 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.415316    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f93f0a1-d934-45f5-9397-98c057e4f050-console-config\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.416639 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.416610    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f93f0a1-d934-45f5-9397-98c057e4f050-console-serving-cert\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.416755 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.416659    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f93f0a1-d934-45f5-9397-98c057e4f050-console-oauth-config\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.427852 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.427816    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl9kc\" (UniqueName: \"kubernetes.io/projected/9f93f0a1-d934-45f5-9397-98c057e4f050-kube-api-access-dl9kc\") pod \"console-577c7d6796-m7ztx\" (UID: \"9f93f0a1-d934-45f5-9397-98c057e4f050\") " pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.532028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.531983    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:34:55.817729 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.817691    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-577c7d6796-m7ztx"]
Oct 30 18:34:55.820608 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:34:55.820572    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f93f0a1_d934_45f5_9397_98c057e4f050.slice/crio-4d3a435bd568837f96842bd1349a583a3138bbf62c66bf6da3a606ffb3af0207 WatchSource:0}: Error finding container 4d3a435bd568837f96842bd1349a583a3138bbf62c66bf6da3a606ffb3af0207: Status 404 returned error can't find the container with id 4d3a435bd568837f96842bd1349a583a3138bbf62c66bf6da3a606ffb3af0207
Oct 30 18:34:55.870224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.870188    4372 generic.go:356] "Generic (PLEG): container finished" podID="83a07661-3916-4026-9853-4edb1210cf2d" containerID="9ae53ffe1324ed5b738301fb4e7ea6e4562fdaa1c98d5bd2a74cefa5af369513" exitCode=0
Oct 30 18:34:55.870763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.870243    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54645595b4-kxm7v"
Oct 30 18:34:55.870763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.870267    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54645595b4-kxm7v" event={"ID":"83a07661-3916-4026-9853-4edb1210cf2d","Type":"ContainerDied","Data":"9ae53ffe1324ed5b738301fb4e7ea6e4562fdaa1c98d5bd2a74cefa5af369513"}
Oct 30 18:34:55.870763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.870341    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54645595b4-kxm7v" event={"ID":"83a07661-3916-4026-9853-4edb1210cf2d","Type":"ContainerDied","Data":"3470436a4d8d86ed6f053f498532f67793e82ecae8ec40eb8afeb9dbd64de538"}
Oct 30 18:34:55.870763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.870373    4372 scope.go:122] "RemoveContainer" containerID="9ae53ffe1324ed5b738301fb4e7ea6e4562fdaa1c98d5bd2a74cefa5af369513"
Oct 30 18:34:55.872119 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.872084    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-577c7d6796-m7ztx" event={"ID":"9f93f0a1-d934-45f5-9397-98c057e4f050","Type":"ContainerStarted","Data":"4d3a435bd568837f96842bd1349a583a3138bbf62c66bf6da3a606ffb3af0207"}
Oct 30 18:34:55.890104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.890078    4372 scope.go:122] "RemoveContainer" containerID="9ae53ffe1324ed5b738301fb4e7ea6e4562fdaa1c98d5bd2a74cefa5af369513"
Oct 30 18:34:55.890569 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:34:55.890538    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae53ffe1324ed5b738301fb4e7ea6e4562fdaa1c98d5bd2a74cefa5af369513\": container with ID starting with 9ae53ffe1324ed5b738301fb4e7ea6e4562fdaa1c98d5bd2a74cefa5af369513 not found: ID does not exist" containerID="9ae53ffe1324ed5b738301fb4e7ea6e4562fdaa1c98d5bd2a74cefa5af369513"
Oct 30 18:34:55.890654 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.890585    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae53ffe1324ed5b738301fb4e7ea6e4562fdaa1c98d5bd2a74cefa5af369513"} err="failed to get container status \"9ae53ffe1324ed5b738301fb4e7ea6e4562fdaa1c98d5bd2a74cefa5af369513\": rpc error: code = NotFound desc = could not find container \"9ae53ffe1324ed5b738301fb4e7ea6e4562fdaa1c98d5bd2a74cefa5af369513\": container with ID starting with 9ae53ffe1324ed5b738301fb4e7ea6e4562fdaa1c98d5bd2a74cefa5af369513 not found: ID does not exist"
Oct 30 18:34:55.903052 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.903006    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54645595b4-kxm7v"]
Oct 30 18:34:55.909878 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:55.909837    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54645595b4-kxm7v"]
Oct 30 18:34:56.263433 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:56.263395    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a07661-3916-4026-9853-4edb1210cf2d" path="/var/lib/kubelet/pods/83a07661-3916-4026-9853-4edb1210cf2d/volumes"
Oct 30 18:34:56.878089 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:56.878045    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-577c7d6796-m7ztx" event={"ID":"9f93f0a1-d934-45f5-9397-98c057e4f050","Type":"ContainerStarted","Data":"5a4390b19a9d5fc8a9483f045cc71fb2f0988f001993cc9bc8a958fb3a74ecf5"}
Oct 30 18:34:56.901016 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:34:56.900936    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-577c7d6796-m7ztx" podStartSLOduration=1.900915745 podStartE2EDuration="1.900915745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:56.900307176 +0000 UTC m=+110.768725505" watchObservedRunningTime="2026-10-30 18:34:56.900915745 +0000 UTC m=+110.769334075"
Oct 30 18:34:57.699815 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:34:57.699761    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:35:05.532962 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:05.532900    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:35:05.533530 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:05.532990    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:35:05.538305 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:05.538280    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:35:05.918906 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:05.918872    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-577c7d6796-m7ztx"
Oct 30 18:35:05.971278 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:05.971238    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cd4c4444f-42tgx"]
Oct 30 18:35:27.699948 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:35:27.699826    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:35:30.993017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:30.992934    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6cd4c4444f-42tgx" podUID="f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8" containerName="console" containerID="cri-o://05540f20452cdeda25d0d883d8575eebf29ca3416052344349bd4bcffc958e58" gracePeriod=15
Oct 30 18:35:31.243694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.243601    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:35:31.315253 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.315198    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-kube-api-access-rb75r\" (UniqueName: \"kubernetes.io/projected/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-kube-api-access-rb75r\") pod \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") "
Oct 30 18:35:31.315447 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.315288    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-oauth-serving-cert\") pod \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") "
Oct 30 18:35:31.315447 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.315398    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-service-ca\") pod \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") "
Oct 30 18:35:31.315593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.315443    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-serving-cert\") pod \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") "
Oct 30 18:35:31.315593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.315503    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-trusted-ca-bundle\") pod \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") "
Oct 30 18:35:31.315593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.315551    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-oauth-config\") pod \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") "
Oct 30 18:35:31.315742 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.315609    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-tmp\") pod \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") "
Oct 30 18:35:31.315742 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.315644    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-config\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-config\") pod \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\" (UID: \"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8\") "
Oct 30 18:35:31.315944 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.315916    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-service-ca" pod "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8" (UID: "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:35:31.316093 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.316040    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-tmp" pod "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8" (UID: "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:35:31.316093 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.316060    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-trusted-ca-bundle" pod "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8" (UID: "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:35:31.316482 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.316218    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-config" pod "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8" (UID: "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:35:31.316482 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.315673    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-oauth-serving-cert" pod "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8" (UID: "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:35:31.318066 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.318037    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-serving-cert" pod "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8" (UID: "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:35:31.318220 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.318068    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-oauth-config" pod "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8" (UID: "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:35:31.318220 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.318074    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-kube-api-access-rb75r" pod "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8" (UID: "f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8"). InnerVolumeSpecName "kube-api-access-rb75r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:35:31.417139 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.417078    4372 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-config\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:35:31.417139 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.417114    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb75r\" (UniqueName: \"kubernetes.io/projected/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-kube-api-access-rb75r\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:35:31.417139 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.417149    4372 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-oauth-serving-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:35:31.417395 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.417160    4372 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:35:31.417395 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.417170    4372 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-serving-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:35:31.417395 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.417178    4372 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-trusted-ca-bundle\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:35:31.417395 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.417187    4372 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-console-oauth-config\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:35:31.417395 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:31.417196    4372 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8-tmp\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:35:32.010145 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:32.010073    4372 generic.go:356] "Generic (PLEG): container finished" podID="f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8" containerID="05540f20452cdeda25d0d883d8575eebf29ca3416052344349bd4bcffc958e58" exitCode=0
Oct 30 18:35:32.010612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:32.010162    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd4c4444f-42tgx"
Oct 30 18:35:32.010612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:32.010169    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd4c4444f-42tgx" event={"ID":"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8","Type":"ContainerDied","Data":"05540f20452cdeda25d0d883d8575eebf29ca3416052344349bd4bcffc958e58"}
Oct 30 18:35:32.010612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:32.010241    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd4c4444f-42tgx" event={"ID":"f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8","Type":"ContainerDied","Data":"5e38ea78f68f196a1dacc68d51c1f5ed5e4b0453a141c3d4384d53ab4776618f"}
Oct 30 18:35:32.010612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:32.010264    4372 scope.go:122] "RemoveContainer" containerID="05540f20452cdeda25d0d883d8575eebf29ca3416052344349bd4bcffc958e58"
Oct 30 18:35:32.020370 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:32.020349    4372 scope.go:122] "RemoveContainer" containerID="05540f20452cdeda25d0d883d8575eebf29ca3416052344349bd4bcffc958e58"
Oct 30 18:35:32.020744 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:35:32.020722    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05540f20452cdeda25d0d883d8575eebf29ca3416052344349bd4bcffc958e58\": container with ID starting with 05540f20452cdeda25d0d883d8575eebf29ca3416052344349bd4bcffc958e58 not found: ID does not exist" containerID="05540f20452cdeda25d0d883d8575eebf29ca3416052344349bd4bcffc958e58"
Oct 30 18:35:32.020836 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:32.020755    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05540f20452cdeda25d0d883d8575eebf29ca3416052344349bd4bcffc958e58"} err="failed to get container status \"05540f20452cdeda25d0d883d8575eebf29ca3416052344349bd4bcffc958e58\": rpc error: code = NotFound desc = could not find container \"05540f20452cdeda25d0d883d8575eebf29ca3416052344349bd4bcffc958e58\": container with ID starting with 05540f20452cdeda25d0d883d8575eebf29ca3416052344349bd4bcffc958e58 not found: ID does not exist"
Oct 30 18:35:32.034795 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:32.034755    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cd4c4444f-42tgx"]
Oct 30 18:35:32.039765 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:32.039716    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cd4c4444f-42tgx"]
Oct 30 18:35:32.263437 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:35:32.263332    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8" path="/var/lib/kubelet/pods/f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8/volumes"
Oct 30 18:35:57.699342 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:35:57.699292    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:36:27.700529 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:36:27.700473    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:36:57.700737 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:36:57.700619    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:36:59.604586 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.604545    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"]
Oct 30 18:36:59.605016 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.604808    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8" containerName="console"
Oct 30 18:36:59.605016 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.604820    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8" containerName="console"
Oct 30 18:36:59.605016 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.604872    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f5ed70-3252-4f2a-8c4b-cb9f7faff5f8" containerName="console"
Oct 30 18:36:59.607973 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.607947    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"
Oct 30 18:36:59.625366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.625319    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nnhv\" (UniqueName: \"kubernetes.io/projected/863f9496-789b-4148-a5e4-47164375c502-kube-api-access-7nnhv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv\" (UID: \"863f9496-789b-4148-a5e4-47164375c502\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"
Oct 30 18:36:59.625617 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.625478    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/863f9496-789b-4148-a5e4-47164375c502-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv\" (UID: \"863f9496-789b-4148-a5e4-47164375c502\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"
Oct 30 18:36:59.625617 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.625515    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.625617 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.625521    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-sz2z9\""
Oct 30 18:36:59.625617 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.625532    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.625617 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.625601    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/863f9496-789b-4148-a5e4-47164375c502-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv\" (UID: \"863f9496-789b-4148-a5e4-47164375c502\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"
Oct 30 18:36:59.627161 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.627114    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"]
Oct 30 18:36:59.697789 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.697747    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"]
Oct 30 18:36:59.701481 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.701453    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"
Oct 30 18:36:59.717779 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.717738    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-vlnc7\""
Oct 30 18:36:59.725988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.725947    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/863f9496-789b-4148-a5e4-47164375c502-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv\" (UID: \"863f9496-789b-4148-a5e4-47164375c502\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"
Oct 30 18:36:59.726271 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.725997    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8732dbd-fa69-40d3-9dd9-9f65694d0435-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g\" (UID: \"f8732dbd-fa69-40d3-9dd9-9f65694d0435\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"
Oct 30 18:36:59.726271 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.726043    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/863f9496-789b-4148-a5e4-47164375c502-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv\" (UID: \"863f9496-789b-4148-a5e4-47164375c502\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"
Oct 30 18:36:59.726271 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.726077    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nnhv\" (UniqueName: \"kubernetes.io/projected/863f9496-789b-4148-a5e4-47164375c502-kube-api-access-7nnhv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv\" (UID: \"863f9496-789b-4148-a5e4-47164375c502\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"
Oct 30 18:36:59.726271 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.726206    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8732dbd-fa69-40d3-9dd9-9f65694d0435-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g\" (UID: \"f8732dbd-fa69-40d3-9dd9-9f65694d0435\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"
Oct 30 18:36:59.726271 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.726258    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l59gz\" (UniqueName: \"kubernetes.io/projected/f8732dbd-fa69-40d3-9dd9-9f65694d0435-kube-api-access-l59gz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g\" (UID: \"f8732dbd-fa69-40d3-9dd9-9f65694d0435\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"
Oct 30 18:36:59.726536 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.726483    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/863f9496-789b-4148-a5e4-47164375c502-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv\" (UID: \"863f9496-789b-4148-a5e4-47164375c502\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"
Oct 30 18:36:59.726536 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.726515    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/863f9496-789b-4148-a5e4-47164375c502-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv\" (UID: \"863f9496-789b-4148-a5e4-47164375c502\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"
Oct 30 18:36:59.726743 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.726725    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"]
Oct 30 18:36:59.742808 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.742317    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nnhv\" (UniqueName: \"kubernetes.io/projected/863f9496-789b-4148-a5e4-47164375c502-kube-api-access-7nnhv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv\" (UID: \"863f9496-789b-4148-a5e4-47164375c502\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"
Oct 30 18:36:59.808761 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.808719    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"]
Oct 30 18:36:59.812300 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.812273    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.819170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.819117    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 18:36:59.823004 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.822976    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 18:36:59.823214 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.823171    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7h4l8\""
Oct 30 18:36:59.824505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.824486    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.824587 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.824544    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 18:36:59.824699 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.824687    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.827258 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.827223    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"]
Oct 30 18:36:59.827515 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.827494    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8732dbd-fa69-40d3-9dd9-9f65694d0435-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g\" (UID: \"f8732dbd-fa69-40d3-9dd9-9f65694d0435\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"
Oct 30 18:36:59.827576 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.827545    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.827617 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.827582    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.827673 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.827617    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8732dbd-fa69-40d3-9dd9-9f65694d0435-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g\" (UID: \"f8732dbd-fa69-40d3-9dd9-9f65694d0435\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"
Oct 30 18:36:59.827673 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.827649    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.827771 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.827740    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r6q6\" (UniqueName: \"kubernetes.io/projected/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-kube-api-access-2r6q6\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.827824 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.827797    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59gz\" (UniqueName: \"kubernetes.io/projected/f8732dbd-fa69-40d3-9dd9-9f65694d0435-kube-api-access-l59gz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g\" (UID: \"f8732dbd-fa69-40d3-9dd9-9f65694d0435\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"
Oct 30 18:36:59.827882 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.827831    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.827935 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.827901    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.827986 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.827953    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.827986 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.827966    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8732dbd-fa69-40d3-9dd9-9f65694d0435-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g\" (UID: \"f8732dbd-fa69-40d3-9dd9-9f65694d0435\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"
Oct 30 18:36:59.828433 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.828401    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8732dbd-fa69-40d3-9dd9-9f65694d0435-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g\" (UID: \"f8732dbd-fa69-40d3-9dd9-9f65694d0435\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"
Oct 30 18:36:59.837020 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.836985    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 18:36:59.853257 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.853221    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l59gz\" (UniqueName: \"kubernetes.io/projected/f8732dbd-fa69-40d3-9dd9-9f65694d0435-kube-api-access-l59gz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g\" (UID: \"f8732dbd-fa69-40d3-9dd9-9f65694d0435\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"
Oct 30 18:36:59.925828 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.925710    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"
Oct 30 18:36:59.928604 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.928567    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.928774 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.928618    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.928774 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.928663    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.928774 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.928690    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.928774 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.928725    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.928774 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.928748    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r6q6\" (UniqueName: \"kubernetes.io/projected/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-kube-api-access-2r6q6\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.929057 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.928778    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.930011 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.929339    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.931933 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.931897    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.932076 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.931927    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.932076 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.931994    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.932294 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.932268    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.961566 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.961528    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:36:59.961731 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:36:59.961546    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r6q6\" (UniqueName: \"kubernetes.io/projected/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-kube-api-access-2r6q6\") pod \"istiod-openshift-gateway-7876bfcb74-l66w5\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:37:00.018459 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:00.018403    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"
Oct 30 18:37:00.126492 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:00.126449    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:37:00.215802 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:00.215754    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"]
Oct 30 18:37:00.219663 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:37:00.219617    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863f9496_789b_4148_a5e4_47164375c502.slice/crio-a32adbdc0227a14677f532c5e1a6bb9a52c62a93823f8617595abff622992f2f WatchSource:0}: Error finding container a32adbdc0227a14677f532c5e1a6bb9a52c62a93823f8617595abff622992f2f: Status 404 returned error can't find the container with id a32adbdc0227a14677f532c5e1a6bb9a52c62a93823f8617595abff622992f2f
Oct 30 18:37:00.307207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:00.307025    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"]
Oct 30 18:37:00.310152 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:37:00.310101    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8732dbd_fa69_40d3_9dd9_9f65694d0435.slice/crio-ba119f69849753b3933f1c46b307619e67859e168b536f5e80b64cd0b9be94f5 WatchSource:0}: Error finding container ba119f69849753b3933f1c46b307619e67859e168b536f5e80b64cd0b9be94f5: Status 404 returned error can't find the container with id ba119f69849753b3933f1c46b307619e67859e168b536f5e80b64cd0b9be94f5
Oct 30 18:37:00.335002 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:00.334942    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv" event={"ID":"863f9496-789b-4148-a5e4-47164375c502","Type":"ContainerStarted","Data":"a32adbdc0227a14677f532c5e1a6bb9a52c62a93823f8617595abff622992f2f"}
Oct 30 18:37:00.336427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:00.336383    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g" event={"ID":"f8732dbd-fa69-40d3-9dd9-9f65694d0435","Type":"ContainerStarted","Data":"ba119f69849753b3933f1c46b307619e67859e168b536f5e80b64cd0b9be94f5"}
Oct 30 18:37:00.464940 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:37:00.464902    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21dd3100_78ff_4e52_8af1_ad4ed7ecd9c9.slice/crio-37d67c38c2af4d01fb7e4f31f8eb4958ba97a4b7035994c3e90a18ff60fb76b4 WatchSource:0}: Error finding container 37d67c38c2af4d01fb7e4f31f8eb4958ba97a4b7035994c3e90a18ff60fb76b4: Status 404 returned error can't find the container with id 37d67c38c2af4d01fb7e4f31f8eb4958ba97a4b7035994c3e90a18ff60fb76b4
Oct 30 18:37:00.474701 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:00.474662    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"]
Oct 30 18:37:01.344255 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:01.344197    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5" event={"ID":"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9","Type":"ContainerStarted","Data":"37d67c38c2af4d01fb7e4f31f8eb4958ba97a4b7035994c3e90a18ff60fb76b4"}
Oct 30 18:37:04.423636 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:04.423590    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:37:04.424172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:04.423719    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:37:05.367299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:05.367248    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5" event={"ID":"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9","Type":"ContainerStarted","Data":"b0e5e54fba06985154d72f57c3df2d5f0125f82605365e639b8c6922c68615e4"}
Oct 30 18:37:05.367782 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:05.367753    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:37:05.370073 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:05.370034    4372 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-l66w5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:37:05.370276 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:05.370115    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5" podUID="21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:37:05.420037 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:05.419940    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5" podStartSLOduration=2.464730971 podStartE2EDuration="6.419910543s" totalImagesPullingTime="3.955179572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:05.41747693 +0000 UTC m=+239.285895264" watchObservedRunningTime="2026-10-30 18:37:05.419910543 +0000 UTC m=+239.288328877"
Oct 30 18:37:06.375154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:06.375094    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:37:08.382438 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:08.382378    4372 generic.go:356] "Generic (PLEG): container finished" podID="f8732dbd-fa69-40d3-9dd9-9f65694d0435" containerID="4dec2b93972640836507509bc8901060aa6f925dbb14fb2898685b7c45a1c3e8" exitCode=0
Oct 30 18:37:08.383048 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:08.382479    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g" event={"ID":"f8732dbd-fa69-40d3-9dd9-9f65694d0435","Type":"ContainerDied","Data":"4dec2b93972640836507509bc8901060aa6f925dbb14fb2898685b7c45a1c3e8"}
Oct 30 18:37:08.384435 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:08.384395    4372 generic.go:356] "Generic (PLEG): container finished" podID="863f9496-789b-4148-a5e4-47164375c502" containerID="9bf9582ae6c84a2f8d91e2a37b9550f1593d5e4d8cbb143c928b37f550ca4e61" exitCode=0
Oct 30 18:37:08.384550 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:08.384511    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv" event={"ID":"863f9496-789b-4148-a5e4-47164375c502","Type":"ContainerDied","Data":"9bf9582ae6c84a2f8d91e2a37b9550f1593d5e4d8cbb143c928b37f550ca4e61"}
Oct 30 18:37:09.391677 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:09.391626    4372 generic.go:356] "Generic (PLEG): container finished" podID="863f9496-789b-4148-a5e4-47164375c502" containerID="7d27fdac39000d2fdcbc91e5a94403a81ffd39a1847e1b8b5bcfa5e66f1bda8d" exitCode=0
Oct 30 18:37:09.392247 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:09.391711    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv" event={"ID":"863f9496-789b-4148-a5e4-47164375c502","Type":"ContainerDied","Data":"7d27fdac39000d2fdcbc91e5a94403a81ffd39a1847e1b8b5bcfa5e66f1bda8d"}
Oct 30 18:37:11.404187 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:11.404113    4372 generic.go:356] "Generic (PLEG): container finished" podID="f8732dbd-fa69-40d3-9dd9-9f65694d0435" containerID="e93b501903684186c445e41e18f498c3cba9cd45ee74a51b0f0364eb65f51a28" exitCode=0
Oct 30 18:37:11.404749 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:11.404201    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g" event={"ID":"f8732dbd-fa69-40d3-9dd9-9f65694d0435","Type":"ContainerDied","Data":"e93b501903684186c445e41e18f498c3cba9cd45ee74a51b0f0364eb65f51a28"}
Oct 30 18:37:16.432110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:16.432056    4372 generic.go:356] "Generic (PLEG): container finished" podID="863f9496-789b-4148-a5e4-47164375c502" containerID="3ba5248559a88dab50f1f43fc1d6bb5cd1d73d2d0d1f4ce6ae76d30dac0800a7" exitCode=0
Oct 30 18:37:16.432708 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:16.432169    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv" event={"ID":"863f9496-789b-4148-a5e4-47164375c502","Type":"ContainerDied","Data":"3ba5248559a88dab50f1f43fc1d6bb5cd1d73d2d0d1f4ce6ae76d30dac0800a7"}
Oct 30 18:37:17.591949 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:17.591904    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"
Oct 30 18:37:17.712479 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:17.712365    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/863f9496-789b-4148-a5e4-47164375c502-kube-api-access-7nnhv\" (UniqueName: \"kubernetes.io/projected/863f9496-789b-4148-a5e4-47164375c502-kube-api-access-7nnhv\") pod \"863f9496-789b-4148-a5e4-47164375c502\" (UID: \"863f9496-789b-4148-a5e4-47164375c502\") "
Oct 30 18:37:17.712479 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:17.712426    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863f9496-789b-4148-a5e4-47164375c502-bundle\" (UniqueName: \"kubernetes.io/empty-dir/863f9496-789b-4148-a5e4-47164375c502-bundle\") pod \"863f9496-789b-4148-a5e4-47164375c502\" (UID: \"863f9496-789b-4148-a5e4-47164375c502\") "
Oct 30 18:37:17.712759 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:17.712485    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863f9496-789b-4148-a5e4-47164375c502-util\" (UniqueName: \"kubernetes.io/empty-dir/863f9496-789b-4148-a5e4-47164375c502-util\") pod \"863f9496-789b-4148-a5e4-47164375c502\" (UID: \"863f9496-789b-4148-a5e4-47164375c502\") "
Oct 30 18:37:17.713333 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:17.713293    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863f9496-789b-4148-a5e4-47164375c502-bundle" pod "863f9496-789b-4148-a5e4-47164375c502" (UID: "863f9496-789b-4148-a5e4-47164375c502"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:17.715722 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:17.715679    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863f9496-789b-4148-a5e4-47164375c502-kube-api-access-7nnhv" pod "863f9496-789b-4148-a5e4-47164375c502" (UID: "863f9496-789b-4148-a5e4-47164375c502"). InnerVolumeSpecName "kube-api-access-7nnhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:17.718082 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:17.718029    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863f9496-789b-4148-a5e4-47164375c502-util" pod "863f9496-789b-4148-a5e4-47164375c502" (UID: "863f9496-789b-4148-a5e4-47164375c502"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:17.813494 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:17.813445    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nnhv\" (UniqueName: \"kubernetes.io/projected/863f9496-789b-4148-a5e4-47164375c502-kube-api-access-7nnhv\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:17.813494 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:17.813489    4372 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/863f9496-789b-4148-a5e4-47164375c502-bundle\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:17.813494 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:17.813503    4372 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/863f9496-789b-4148-a5e4-47164375c502-util\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:18.444249 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:18.444139    4372 generic.go:356] "Generic (PLEG): container finished" podID="f8732dbd-fa69-40d3-9dd9-9f65694d0435" containerID="1670950539f70e8a1f570a876b68a188020f48b36670c14ba4c62eef56a86e04" exitCode=0
Oct 30 18:37:18.444484 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:18.444236    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g" event={"ID":"f8732dbd-fa69-40d3-9dd9-9f65694d0435","Type":"ContainerDied","Data":"1670950539f70e8a1f570a876b68a188020f48b36670c14ba4c62eef56a86e04"}
Oct 30 18:37:18.446405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:18.446378    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv" event={"ID":"863f9496-789b-4148-a5e4-47164375c502","Type":"ContainerDied","Data":"a32adbdc0227a14677f532c5e1a6bb9a52c62a93823f8617595abff622992f2f"}
Oct 30 18:37:18.446405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:18.446396    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nlwgv"
Oct 30 18:37:18.446614 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:18.446425    4372 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a32adbdc0227a14677f532c5e1a6bb9a52c62a93823f8617595abff622992f2f"
Oct 30 18:37:19.589845 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:19.589801    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"
Oct 30 18:37:19.728921 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:19.728790    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8732dbd-fa69-40d3-9dd9-9f65694d0435-util\" (UniqueName: \"kubernetes.io/empty-dir/f8732dbd-fa69-40d3-9dd9-9f65694d0435-util\") pod \"f8732dbd-fa69-40d3-9dd9-9f65694d0435\" (UID: \"f8732dbd-fa69-40d3-9dd9-9f65694d0435\") "
Oct 30 18:37:19.728921 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:19.728847    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8732dbd-fa69-40d3-9dd9-9f65694d0435-kube-api-access-l59gz\" (UniqueName: \"kubernetes.io/projected/f8732dbd-fa69-40d3-9dd9-9f65694d0435-kube-api-access-l59gz\") pod \"f8732dbd-fa69-40d3-9dd9-9f65694d0435\" (UID: \"f8732dbd-fa69-40d3-9dd9-9f65694d0435\") "
Oct 30 18:37:19.729271 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:19.728932    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8732dbd-fa69-40d3-9dd9-9f65694d0435-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8732dbd-fa69-40d3-9dd9-9f65694d0435-bundle\") pod \"f8732dbd-fa69-40d3-9dd9-9f65694d0435\" (UID: \"f8732dbd-fa69-40d3-9dd9-9f65694d0435\") "
Oct 30 18:37:19.729474 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:19.729436    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8732dbd-fa69-40d3-9dd9-9f65694d0435-bundle" pod "f8732dbd-fa69-40d3-9dd9-9f65694d0435" (UID: "f8732dbd-fa69-40d3-9dd9-9f65694d0435"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:19.731684 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:19.731649    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8732dbd-fa69-40d3-9dd9-9f65694d0435-kube-api-access-l59gz" pod "f8732dbd-fa69-40d3-9dd9-9f65694d0435" (UID: "f8732dbd-fa69-40d3-9dd9-9f65694d0435"). InnerVolumeSpecName "kube-api-access-l59gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:19.735494 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:19.735446    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8732dbd-fa69-40d3-9dd9-9f65694d0435-util" pod "f8732dbd-fa69-40d3-9dd9-9f65694d0435" (UID: "f8732dbd-fa69-40d3-9dd9-9f65694d0435"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:19.830067 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:19.830012    4372 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8732dbd-fa69-40d3-9dd9-9f65694d0435-bundle\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:19.830067 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:19.830051    4372 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8732dbd-fa69-40d3-9dd9-9f65694d0435-util\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:19.830067 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:19.830062    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l59gz\" (UniqueName: \"kubernetes.io/projected/f8732dbd-fa69-40d3-9dd9-9f65694d0435-kube-api-access-l59gz\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:20.456505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:20.456471    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g"
Oct 30 18:37:20.456756 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:20.456468    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tkc4g" event={"ID":"f8732dbd-fa69-40d3-9dd9-9f65694d0435","Type":"ContainerDied","Data":"ba119f69849753b3933f1c46b307619e67859e168b536f5e80b64cd0b9be94f5"}
Oct 30 18:37:20.456756 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:20.456605    4372 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba119f69849753b3933f1c46b307619e67859e168b536f5e80b64cd0b9be94f5"
Oct 30 18:37:27.700014 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:37:27.699968    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:37:41.608498 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.608449    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"]
Oct 30 18:37:41.609017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.608782    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f8732dbd-fa69-40d3-9dd9-9f65694d0435" containerName="util"
Oct 30 18:37:41.609017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.608794    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8732dbd-fa69-40d3-9dd9-9f65694d0435" containerName="util"
Oct 30 18:37:41.609017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.608805    4372 cpu_manager.go:414] "containerMap: removing container" podUID="863f9496-789b-4148-a5e4-47164375c502" containerName="pull"
Oct 30 18:37:41.609017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.608809    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="863f9496-789b-4148-a5e4-47164375c502" containerName="pull"
Oct 30 18:37:41.609017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.608822    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f8732dbd-fa69-40d3-9dd9-9f65694d0435" containerName="extract"
Oct 30 18:37:41.609017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.608827    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8732dbd-fa69-40d3-9dd9-9f65694d0435" containerName="extract"
Oct 30 18:37:41.609017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.608837    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f8732dbd-fa69-40d3-9dd9-9f65694d0435" containerName="pull"
Oct 30 18:37:41.609017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.608842    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8732dbd-fa69-40d3-9dd9-9f65694d0435" containerName="pull"
Oct 30 18:37:41.609017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.608848    4372 cpu_manager.go:414] "containerMap: removing container" podUID="863f9496-789b-4148-a5e4-47164375c502" containerName="extract"
Oct 30 18:37:41.609017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.608853    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="863f9496-789b-4148-a5e4-47164375c502" containerName="extract"
Oct 30 18:37:41.609017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.608864    4372 cpu_manager.go:414] "containerMap: removing container" podUID="863f9496-789b-4148-a5e4-47164375c502" containerName="util"
Oct 30 18:37:41.609017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.608868    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="863f9496-789b-4148-a5e4-47164375c502" containerName="util"
Oct 30 18:37:41.609017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.608917    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="863f9496-789b-4148-a5e4-47164375c502" containerName="extract"
Oct 30 18:37:41.609017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.608926    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8732dbd-fa69-40d3-9dd9-9f65694d0435" containerName="extract"
Oct 30 18:37:41.611757 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.611731    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.641172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.641109    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 18:37:41.641172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.641184    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 18:37:41.681612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.681561    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"]
Oct 30 18:37:41.719341 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.719304    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 18:37:41.719623 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.719584    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 18:37:41.719742 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.719721    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-c242x\""
Oct 30 18:37:41.719792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.719724    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 18:37:41.722211 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.722082    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skdff\" (UniqueName: \"kubernetes.io/projected/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-kube-api-access-skdff\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.722294 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.722232    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-tmp\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.722294 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.722274    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.722358 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.722337    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.722413 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.722397    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-manifests\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.722457 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.722445    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-metrics-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.823977 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.823936    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-manifests\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.824299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.823992    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-metrics-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.824299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.824034    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skdff\" (UniqueName: \"kubernetes.io/projected/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-kube-api-access-skdff\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.824299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.824058    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-tmp\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.824299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.824087    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.824299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.824115    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.824578 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.824547    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-manifests\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.827112 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.827070    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-tmp\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.827394 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.827371    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-metrics-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.827501 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.827477    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.827544 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.827477    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.861409 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.861325    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skdff\" (UniqueName: \"kubernetes.io/projected/f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6-kube-api-access-skdff\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-n9hh6\" (UID: \"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:41.927204 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:41.927151    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:42.287360 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:42.287328    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"]
Oct 30 18:37:42.290713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:37:42.290665    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ace80b_d8a9_4fd1_a6cf_3d67475fe9d6.slice/crio-9748f922f8bbd1c9a4aab37c2878f852c5607f0fa46a1017a285383ed112012d WatchSource:0}: Error finding container 9748f922f8bbd1c9a4aab37c2878f852c5607f0fa46a1017a285383ed112012d: Status 404 returned error can't find the container with id 9748f922f8bbd1c9a4aab37c2878f852c5607f0fa46a1017a285383ed112012d
Oct 30 18:37:42.541613 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:42.541499    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6" event={"ID":"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6","Type":"ContainerStarted","Data":"9748f922f8bbd1c9a4aab37c2878f852c5607f0fa46a1017a285383ed112012d"}
Oct 30 18:37:45.555108 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:45.554998    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6" event={"ID":"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6","Type":"ContainerStarted","Data":"1dcb590d18e7c2c29c61a99ad15eef30063894a92f2749ccabcc12d8a7b6054b"}
Oct 30 18:37:46.561344 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:46.561289    4372 generic.go:356] "Generic (PLEG): container finished" podID="f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6" containerID="1dcb590d18e7c2c29c61a99ad15eef30063894a92f2749ccabcc12d8a7b6054b" exitCode=0
Oct 30 18:37:46.561825 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:46.561359    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6" event={"ID":"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6","Type":"ContainerDied","Data":"1dcb590d18e7c2c29c61a99ad15eef30063894a92f2749ccabcc12d8a7b6054b"}
Oct 30 18:37:47.566874 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:47.566829    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6" event={"ID":"f9ace80b-d8a9-4fd1-a6cf-3d67475fe9d6","Type":"ContainerStarted","Data":"f044350699445d6be889b223f8ef9a28ad9460f71f69d6c2f080e3875f876dde"}
Oct 30 18:37:47.567374 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:47.566951    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:37:47.736429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:47.736359    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6" podStartSLOduration=2.723933234 podStartE2EDuration="6.736336227s" totalImagesPullingTime="3.464082513s" totalInitContainerRuntime="548.32048ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:47.735167917 +0000 UTC m=+281.603586248" watchObservedRunningTime="2026-10-30 18:37:47.736336227 +0000 UTC m=+281.604754710"
Oct 30 18:37:57.699670 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:37:57.699623    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:37:58.573369 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:37:58.573333    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-n9hh6"
Oct 30 18:38:05.849794 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:05.849738    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-zjgtb"]
Oct 30 18:38:05.856277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:05.856238    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zjgtb"
Oct 30 18:38:05.890020 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:05.889963    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 18:38:05.890020 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:05.889964    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 18:38:05.891670 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:05.891645    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-pw7d8\""
Oct 30 18:38:05.896411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:05.896374    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-zjgtb"]
Oct 30 18:38:05.935671 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:05.935606    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73d64847-c779-45af-95d3-00f700291641-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-zjgtb\" (UID: \"73d64847-c779-45af-95d3-00f700291641\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zjgtb"
Oct 30 18:38:05.935671 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:05.935681    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzgkx\" (UniqueName: \"kubernetes.io/projected/73d64847-c779-45af-95d3-00f700291641-kube-api-access-hzgkx\") pod \"cert-manager-cainjector-5c57bbd4fc-zjgtb\" (UID: \"73d64847-c779-45af-95d3-00f700291641\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zjgtb"
Oct 30 18:38:06.036736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:06.036664    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73d64847-c779-45af-95d3-00f700291641-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-zjgtb\" (UID: \"73d64847-c779-45af-95d3-00f700291641\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zjgtb"
Oct 30 18:38:06.036736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:06.036737    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzgkx\" (UniqueName: \"kubernetes.io/projected/73d64847-c779-45af-95d3-00f700291641-kube-api-access-hzgkx\") pod \"cert-manager-cainjector-5c57bbd4fc-zjgtb\" (UID: \"73d64847-c779-45af-95d3-00f700291641\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zjgtb"
Oct 30 18:38:06.055047 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:06.054997    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73d64847-c779-45af-95d3-00f700291641-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-zjgtb\" (UID: \"73d64847-c779-45af-95d3-00f700291641\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zjgtb"
Oct 30 18:38:06.055297 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:06.055075    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzgkx\" (UniqueName: \"kubernetes.io/projected/73d64847-c779-45af-95d3-00f700291641-kube-api-access-hzgkx\") pod \"cert-manager-cainjector-5c57bbd4fc-zjgtb\" (UID: \"73d64847-c779-45af-95d3-00f700291641\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zjgtb"
Oct 30 18:38:06.187618 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:06.187478    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zjgtb"
Oct 30 18:38:06.498033 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:06.497920    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-zjgtb"]
Oct 30 18:38:06.502386 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:38:06.502342    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73d64847_c779_45af_95d3_00f700291641.slice/crio-e0ca34823da71f023c1d368b47c4c38f009c78ff466cd8976aac1c46efcafca4 WatchSource:0}: Error finding container e0ca34823da71f023c1d368b47c4c38f009c78ff466cd8976aac1c46efcafca4: Status 404 returned error can't find the container with id e0ca34823da71f023c1d368b47c4c38f009c78ff466cd8976aac1c46efcafca4
Oct 30 18:38:06.647351 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:06.647304    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zjgtb" event={"ID":"73d64847-c779-45af-95d3-00f700291641","Type":"ContainerStarted","Data":"e0ca34823da71f023c1d368b47c4c38f009c78ff466cd8976aac1c46efcafca4"}
Oct 30 18:38:10.668621 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:10.668567    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zjgtb" event={"ID":"73d64847-c779-45af-95d3-00f700291641","Type":"ContainerStarted","Data":"08e489e6c2c6da097d0fabf997cf71eaa1a2c7e28121f7ece711069f0d989fe4"}
Oct 30 18:38:10.721663 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:10.721559    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zjgtb" podStartSLOduration=2.349084248 podStartE2EDuration="5.721536756s" totalImagesPullingTime="3.372452508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:10.720076452 +0000 UTC m=+304.588494782" watchObservedRunningTime="2026-10-30 18:38:10.721536756 +0000 UTC m=+304.589955085"
Oct 30 18:38:13.112156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:13.112075    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-q8vrg"]
Oct 30 18:38:13.115914 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:13.115878    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-q8vrg"
Oct 30 18:38:13.160576 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:13.160517    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-q8vrg"]
Oct 30 18:38:13.207873 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:13.207811    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/521b401b-547d-4e66-8bf0-ed6ee12b0d68-bound-sa-token\") pod \"cert-manager-597b95cb84-q8vrg\" (UID: \"521b401b-547d-4e66-8bf0-ed6ee12b0d68\") " pod="cert-manager/cert-manager-597b95cb84-q8vrg"
Oct 30 18:38:13.208116 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:13.207900    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22s5z\" (UniqueName: \"kubernetes.io/projected/521b401b-547d-4e66-8bf0-ed6ee12b0d68-kube-api-access-22s5z\") pod \"cert-manager-597b95cb84-q8vrg\" (UID: \"521b401b-547d-4e66-8bf0-ed6ee12b0d68\") " pod="cert-manager/cert-manager-597b95cb84-q8vrg"
Oct 30 18:38:13.309077 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:13.309018    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22s5z\" (UniqueName: \"kubernetes.io/projected/521b401b-547d-4e66-8bf0-ed6ee12b0d68-kube-api-access-22s5z\") pod \"cert-manager-597b95cb84-q8vrg\" (UID: \"521b401b-547d-4e66-8bf0-ed6ee12b0d68\") " pod="cert-manager/cert-manager-597b95cb84-q8vrg"
Oct 30 18:38:13.309330 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:13.309099    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/521b401b-547d-4e66-8bf0-ed6ee12b0d68-bound-sa-token\") pod \"cert-manager-597b95cb84-q8vrg\" (UID: \"521b401b-547d-4e66-8bf0-ed6ee12b0d68\") " pod="cert-manager/cert-manager-597b95cb84-q8vrg"
Oct 30 18:38:13.388426 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:13.388325    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/521b401b-547d-4e66-8bf0-ed6ee12b0d68-bound-sa-token\") pod \"cert-manager-597b95cb84-q8vrg\" (UID: \"521b401b-547d-4e66-8bf0-ed6ee12b0d68\") " pod="cert-manager/cert-manager-597b95cb84-q8vrg"
Oct 30 18:38:13.389778 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:13.389752    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22s5z\" (UniqueName: \"kubernetes.io/projected/521b401b-547d-4e66-8bf0-ed6ee12b0d68-kube-api-access-22s5z\") pod \"cert-manager-597b95cb84-q8vrg\" (UID: \"521b401b-547d-4e66-8bf0-ed6ee12b0d68\") " pod="cert-manager/cert-manager-597b95cb84-q8vrg"
Oct 30 18:38:13.433837 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:13.433788    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-q8vrg"
Oct 30 18:38:13.792431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:13.792343    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-q8vrg"]
Oct 30 18:38:13.796750 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:38:13.796715    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod521b401b_547d_4e66_8bf0_ed6ee12b0d68.slice/crio-7dbdcbbdc0bb0af85a90aa41de777a1845de60ab8d472422732c1e068050a5dd WatchSource:0}: Error finding container 7dbdcbbdc0bb0af85a90aa41de777a1845de60ab8d472422732c1e068050a5dd: Status 404 returned error can't find the container with id 7dbdcbbdc0bb0af85a90aa41de777a1845de60ab8d472422732c1e068050a5dd
Oct 30 18:38:14.689448 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:14.689396    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-q8vrg" event={"ID":"521b401b-547d-4e66-8bf0-ed6ee12b0d68","Type":"ContainerStarted","Data":"966bee1036a8178652da02d73ebd1af510b1f1ecdf528f2c3b3d069e6d15a46a"}
Oct 30 18:38:14.689448 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:14.689455    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-q8vrg" event={"ID":"521b401b-547d-4e66-8bf0-ed6ee12b0d68","Type":"ContainerStarted","Data":"7dbdcbbdc0bb0af85a90aa41de777a1845de60ab8d472422732c1e068050a5dd"}
Oct 30 18:38:14.770068 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:14.769989    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-q8vrg" podStartSLOduration=1.769972834 podStartE2EDuration="1.769972834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:14.768729481 +0000 UTC m=+308.637147837" watchObservedRunningTime="2026-10-30 18:38:14.769972834 +0000 UTC m=+308.638391164"
Oct 30 18:38:14.771943 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:14.771911    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-57l2k"]
Oct 30 18:38:14.775425 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:14.775399    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k"
Oct 30 18:38:14.796353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:14.796317    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 18:38:14.796353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:14.796322    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 18:38:14.796586 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:14.796374    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 18:38:14.800009 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:14.799967    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-57l2k"]
Oct 30 18:38:14.853199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:14.853157    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hdz2z\""
Oct 30 18:38:14.886485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:14.886438    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-57l2k"]
Oct 30 18:38:14.924383 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:14.924338    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zflqq\" (UniqueName: \"kubernetes.io/projected/0736f3ca-42d7-4f05-9f58-64664a213a0c-kube-api-access-zflqq\") pod \"limitador-limitador-74dfcfb894-57l2k\" (UID: \"0736f3ca-42d7-4f05-9f58-64664a213a0c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k"
Oct 30 18:38:14.924601 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:14.924398    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0736f3ca-42d7-4f05-9f58-64664a213a0c-config-file\") pod \"limitador-limitador-74dfcfb894-57l2k\" (UID: \"0736f3ca-42d7-4f05-9f58-64664a213a0c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k"
Oct 30 18:38:15.025939 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:15.025824    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zflqq\" (UniqueName: \"kubernetes.io/projected/0736f3ca-42d7-4f05-9f58-64664a213a0c-kube-api-access-zflqq\") pod \"limitador-limitador-74dfcfb894-57l2k\" (UID: \"0736f3ca-42d7-4f05-9f58-64664a213a0c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k"
Oct 30 18:38:15.025939 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:15.025888    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0736f3ca-42d7-4f05-9f58-64664a213a0c-config-file\") pod \"limitador-limitador-74dfcfb894-57l2k\" (UID: \"0736f3ca-42d7-4f05-9f58-64664a213a0c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k"
Oct 30 18:38:15.026627 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:15.026603    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0736f3ca-42d7-4f05-9f58-64664a213a0c-config-file\") pod \"limitador-limitador-74dfcfb894-57l2k\" (UID: \"0736f3ca-42d7-4f05-9f58-64664a213a0c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k"
Oct 30 18:38:15.041899 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:15.041846    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zflqq\" (UniqueName: \"kubernetes.io/projected/0736f3ca-42d7-4f05-9f58-64664a213a0c-kube-api-access-zflqq\") pod \"limitador-limitador-74dfcfb894-57l2k\" (UID: \"0736f3ca-42d7-4f05-9f58-64664a213a0c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k"
Oct 30 18:38:15.090605 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:15.090554    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k"
Oct 30 18:38:15.386366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:15.386279    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-57l2k"]
Oct 30 18:38:15.389520 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:38:15.389475    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0736f3ca_42d7_4f05_9f58_64664a213a0c.slice/crio-2a7514d7a96789b37d7c5e4a687bf70d5a128462e6e1748f4331204e9366ccaa WatchSource:0}: Error finding container 2a7514d7a96789b37d7c5e4a687bf70d5a128462e6e1748f4331204e9366ccaa: Status 404 returned error can't find the container with id 2a7514d7a96789b37d7c5e4a687bf70d5a128462e6e1748f4331204e9366ccaa
Oct 30 18:38:15.695698 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:15.695565    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k" event={"ID":"0736f3ca-42d7-4f05-9f58-64664a213a0c","Type":"ContainerStarted","Data":"2a7514d7a96789b37d7c5e4a687bf70d5a128462e6e1748f4331204e9366ccaa"}
Oct 30 18:38:16.436516 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:16.436431    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:38:16.441326 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:16.441283    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"
Oct 30 18:38:16.471533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:16.471460    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:38:16.496436 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:16.496377    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:38:16.540822 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:16.540768    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/641f9bfc-534b-4a1b-a4de-e68a140ee1f6-config-file\") pod \"limitador-limitador-7c9b87d574-qz7ks\" (UID: \"641f9bfc-534b-4a1b-a4de-e68a140ee1f6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"
Oct 30 18:38:16.541052 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:16.540834    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgkkw\" (UniqueName: \"kubernetes.io/projected/641f9bfc-534b-4a1b-a4de-e68a140ee1f6-kube-api-access-lgkkw\") pod \"limitador-limitador-7c9b87d574-qz7ks\" (UID: \"641f9bfc-534b-4a1b-a4de-e68a140ee1f6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"
Oct 30 18:38:16.642215 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:16.642172    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/641f9bfc-534b-4a1b-a4de-e68a140ee1f6-config-file\") pod \"limitador-limitador-7c9b87d574-qz7ks\" (UID: \"641f9bfc-534b-4a1b-a4de-e68a140ee1f6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"
Oct 30 18:38:16.642467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:16.642224    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgkkw\" (UniqueName: \"kubernetes.io/projected/641f9bfc-534b-4a1b-a4de-e68a140ee1f6-kube-api-access-lgkkw\") pod \"limitador-limitador-7c9b87d574-qz7ks\" (UID: \"641f9bfc-534b-4a1b-a4de-e68a140ee1f6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"
Oct 30 18:38:16.643099 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:16.643062    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/641f9bfc-534b-4a1b-a4de-e68a140ee1f6-config-file\") pod \"limitador-limitador-7c9b87d574-qz7ks\" (UID: \"641f9bfc-534b-4a1b-a4de-e68a140ee1f6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"
Oct 30 18:38:16.695512 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:16.695413    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgkkw\" (UniqueName: \"kubernetes.io/projected/641f9bfc-534b-4a1b-a4de-e68a140ee1f6-kube-api-access-lgkkw\") pod \"limitador-limitador-7c9b87d574-qz7ks\" (UID: \"641f9bfc-534b-4a1b-a4de-e68a140ee1f6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"
Oct 30 18:38:16.762534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:16.762486    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"
Oct 30 18:38:17.104661 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:17.104606    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:38:17.111217 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:38:17.111168    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641f9bfc_534b_4a1b_a4de_e68a140ee1f6.slice/crio-d28765b5f2d396a4acc28154dc348bbf20f39bb95cefc71529b85c96b5bcdfcf WatchSource:0}: Error finding container d28765b5f2d396a4acc28154dc348bbf20f39bb95cefc71529b85c96b5bcdfcf: Status 404 returned error can't find the container with id d28765b5f2d396a4acc28154dc348bbf20f39bb95cefc71529b85c96b5bcdfcf
Oct 30 18:38:17.708716 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:17.708663    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-qz7ks" event={"ID":"641f9bfc-534b-4a1b-a4de-e68a140ee1f6","Type":"ContainerStarted","Data":"d28765b5f2d396a4acc28154dc348bbf20f39bb95cefc71529b85c96b5bcdfcf"}
Oct 30 18:38:18.716276 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:18.716235    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-qz7ks" event={"ID":"641f9bfc-534b-4a1b-a4de-e68a140ee1f6","Type":"ContainerStarted","Data":"278b674f7273955e26837c6372a960a7acfcd72abc98ff02beedbd67a07778a8"}
Oct 30 18:38:18.716784 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:18.716348    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"
Oct 30 18:38:18.718065 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:18.718034    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k" event={"ID":"0736f3ca-42d7-4f05-9f58-64664a213a0c","Type":"ContainerStarted","Data":"ccca544dbca0af74d0f0fc66b737c1351c8b0402127d76c727756346e1bc3e51"}
Oct 30 18:38:18.718230 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:18.718109    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k"
Oct 30 18:38:18.763179 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:18.763073    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-qz7ks" podStartSLOduration=1.339820237 podStartE2EDuration="2.763048348s" totalImagesPullingTime="1.423228111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:18.761957307 +0000 UTC m=+312.630375636" watchObservedRunningTime="2026-10-30 18:38:18.763048348 +0000 UTC m=+312.631466679"
Oct 30 18:38:27.699795 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:38:27.699751    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:38:29.723440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:29.723402    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k"
Oct 30 18:38:29.724046 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:29.723495    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"
Oct 30 18:38:29.747158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:29.747058    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k" podStartSLOduration=12.883292098 podStartE2EDuration="15.747038684s" totalImagesPullingTime="2.863746586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:18.81232816 +0000 UTC m=+312.680746489" watchObservedRunningTime="2026-10-30 18:38:29.747038684 +0000 UTC m=+323.615457014"
Oct 30 18:38:29.817093 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:29.817043    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-57l2k"]
Oct 30 18:38:29.817378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:29.817309    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k" podUID="0736f3ca-42d7-4f05-9f58-64664a213a0c" containerName="limitador" containerID="cri-o://ccca544dbca0af74d0f0fc66b737c1351c8b0402127d76c727756346e1bc3e51" gracePeriod=30
Oct 30 18:38:30.371543 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.371508    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k"
Oct 30 18:38:30.465624 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.465513    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0736f3ca-42d7-4f05-9f58-64664a213a0c-kube-api-access-zflqq\" (UniqueName: \"kubernetes.io/projected/0736f3ca-42d7-4f05-9f58-64664a213a0c-kube-api-access-zflqq\") pod \"0736f3ca-42d7-4f05-9f58-64664a213a0c\" (UID: \"0736f3ca-42d7-4f05-9f58-64664a213a0c\") "
Oct 30 18:38:30.465624 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.465620    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0736f3ca-42d7-4f05-9f58-64664a213a0c-config-file\" (UniqueName: \"kubernetes.io/configmap/0736f3ca-42d7-4f05-9f58-64664a213a0c-config-file\") pod \"0736f3ca-42d7-4f05-9f58-64664a213a0c\" (UID: \"0736f3ca-42d7-4f05-9f58-64664a213a0c\") "
Oct 30 18:38:30.466031 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.466000    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0736f3ca-42d7-4f05-9f58-64664a213a0c-config-file" pod "0736f3ca-42d7-4f05-9f58-64664a213a0c" (UID: "0736f3ca-42d7-4f05-9f58-64664a213a0c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:30.468428 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.468391    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0736f3ca-42d7-4f05-9f58-64664a213a0c-kube-api-access-zflqq" pod "0736f3ca-42d7-4f05-9f58-64664a213a0c" (UID: "0736f3ca-42d7-4f05-9f58-64664a213a0c"). InnerVolumeSpecName "kube-api-access-zflqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:38:30.566379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.566340    4372 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0736f3ca-42d7-4f05-9f58-64664a213a0c-config-file\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:30.566379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.566374    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zflqq\" (UniqueName: \"kubernetes.io/projected/0736f3ca-42d7-4f05-9f58-64664a213a0c-kube-api-access-zflqq\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:30.614213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.614156    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"]
Oct 30 18:38:30.614509 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.614494    4372 cpu_manager.go:414] "containerMap: removing container" podUID="0736f3ca-42d7-4f05-9f58-64664a213a0c" containerName="limitador"
Oct 30 18:38:30.614548 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.614511    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0736f3ca-42d7-4f05-9f58-64664a213a0c" containerName="limitador"
Oct 30 18:38:30.614582 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.614565    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="0736f3ca-42d7-4f05-9f58-64664a213a0c" containerName="limitador"
Oct 30 18:38:30.617807 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.617775    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:30.633931 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.633896    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-lm5tr\""
Oct 30 18:38:30.634151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.633984    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 18:38:30.634151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.634055    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 18:38:30.640077 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.640040    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"]
Oct 30 18:38:30.651750 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.651705    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 18:38:30.651750 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.651705    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 18:38:30.652035 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.651770    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:38:30.667372 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.667330    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvdnx\" (UniqueName: \"kubernetes.io/projected/a9f3ac74-331f-434d-8a21-a47cc078849c-kube-api-access-dvdnx\") pod \"jobset-controller-manager-849b68f5-dflvs\" (UID: \"a9f3ac74-331f-434d-8a21-a47cc078849c\") " pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:30.667576 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.667398    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9f3ac74-331f-434d-8a21-a47cc078849c-metrics-certs\") pod \"jobset-controller-manager-849b68f5-dflvs\" (UID: \"a9f3ac74-331f-434d-8a21-a47cc078849c\") " pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:30.667576 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.667495    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9f3ac74-331f-434d-8a21-a47cc078849c-cert\") pod \"jobset-controller-manager-849b68f5-dflvs\" (UID: \"a9f3ac74-331f-434d-8a21-a47cc078849c\") " pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:30.667576 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.667544    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a9f3ac74-331f-434d-8a21-a47cc078849c-manager-config\") pod \"jobset-controller-manager-849b68f5-dflvs\" (UID: \"a9f3ac74-331f-434d-8a21-a47cc078849c\") " pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:30.768209 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.768100    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9f3ac74-331f-434d-8a21-a47cc078849c-cert\") pod \"jobset-controller-manager-849b68f5-dflvs\" (UID: \"a9f3ac74-331f-434d-8a21-a47cc078849c\") " pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:30.768209 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.768185    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a9f3ac74-331f-434d-8a21-a47cc078849c-manager-config\") pod \"jobset-controller-manager-849b68f5-dflvs\" (UID: \"a9f3ac74-331f-434d-8a21-a47cc078849c\") " pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:30.768742 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.768240    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvdnx\" (UniqueName: \"kubernetes.io/projected/a9f3ac74-331f-434d-8a21-a47cc078849c-kube-api-access-dvdnx\") pod \"jobset-controller-manager-849b68f5-dflvs\" (UID: \"a9f3ac74-331f-434d-8a21-a47cc078849c\") " pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:30.768742 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.768294    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9f3ac74-331f-434d-8a21-a47cc078849c-metrics-certs\") pod \"jobset-controller-manager-849b68f5-dflvs\" (UID: \"a9f3ac74-331f-434d-8a21-a47cc078849c\") " pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:30.769391 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.769359    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a9f3ac74-331f-434d-8a21-a47cc078849c-manager-config\") pod \"jobset-controller-manager-849b68f5-dflvs\" (UID: \"a9f3ac74-331f-434d-8a21-a47cc078849c\") " pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:30.770790 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.770755    4372 generic.go:356] "Generic (PLEG): container finished" podID="0736f3ca-42d7-4f05-9f58-64664a213a0c" containerID="ccca544dbca0af74d0f0fc66b737c1351c8b0402127d76c727756346e1bc3e51" exitCode=0
Oct 30 18:38:30.770926 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.770839    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k"
Oct 30 18:38:30.770926 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.770844    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k" event={"ID":"0736f3ca-42d7-4f05-9f58-64664a213a0c","Type":"ContainerDied","Data":"ccca544dbca0af74d0f0fc66b737c1351c8b0402127d76c727756346e1bc3e51"}
Oct 30 18:38:30.771053 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.770933    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-57l2k" event={"ID":"0736f3ca-42d7-4f05-9f58-64664a213a0c","Type":"ContainerDied","Data":"2a7514d7a96789b37d7c5e4a687bf70d5a128462e6e1748f4331204e9366ccaa"}
Oct 30 18:38:30.771053 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.770981    4372 scope.go:122] "RemoveContainer" containerID="ccca544dbca0af74d0f0fc66b737c1351c8b0402127d76c727756346e1bc3e51"
Oct 30 18:38:30.771688 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.771657    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9f3ac74-331f-434d-8a21-a47cc078849c-cert\") pod \"jobset-controller-manager-849b68f5-dflvs\" (UID: \"a9f3ac74-331f-434d-8a21-a47cc078849c\") " pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:30.771793 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.771705    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9f3ac74-331f-434d-8a21-a47cc078849c-metrics-certs\") pod \"jobset-controller-manager-849b68f5-dflvs\" (UID: \"a9f3ac74-331f-434d-8a21-a47cc078849c\") " pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:30.784502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.784481    4372 scope.go:122] "RemoveContainer" containerID="ccca544dbca0af74d0f0fc66b737c1351c8b0402127d76c727756346e1bc3e51"
Oct 30 18:38:30.784897 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:38:30.784871    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccca544dbca0af74d0f0fc66b737c1351c8b0402127d76c727756346e1bc3e51\": container with ID starting with ccca544dbca0af74d0f0fc66b737c1351c8b0402127d76c727756346e1bc3e51 not found: ID does not exist" containerID="ccca544dbca0af74d0f0fc66b737c1351c8b0402127d76c727756346e1bc3e51"
Oct 30 18:38:30.785007 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.784906    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccca544dbca0af74d0f0fc66b737c1351c8b0402127d76c727756346e1bc3e51"} err="failed to get container status \"ccca544dbca0af74d0f0fc66b737c1351c8b0402127d76c727756346e1bc3e51\": rpc error: code = NotFound desc = could not find container \"ccca544dbca0af74d0f0fc66b737c1351c8b0402127d76c727756346e1bc3e51\": container with ID starting with ccca544dbca0af74d0f0fc66b737c1351c8b0402127d76c727756346e1bc3e51 not found: ID does not exist"
Oct 30 18:38:30.797823 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.797784    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvdnx\" (UniqueName: \"kubernetes.io/projected/a9f3ac74-331f-434d-8a21-a47cc078849c-kube-api-access-dvdnx\") pod \"jobset-controller-manager-849b68f5-dflvs\" (UID: \"a9f3ac74-331f-434d-8a21-a47cc078849c\") " pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:30.815921 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.815877    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-57l2k"]
Oct 30 18:38:30.822555 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.822521    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-57l2k"]
Oct 30 18:38:30.931267 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:30.931215    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:31.231848 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:31.231806    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"]
Oct 30 18:38:31.235325 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:38:31.235286    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f3ac74_331f_434d_8a21_a47cc078849c.slice/crio-8429a488e310d4c2bffbe63d99a31569dbb572abf4c04fe1c91025c4eee799ac WatchSource:0}: Error finding container 8429a488e310d4c2bffbe63d99a31569dbb572abf4c04fe1c91025c4eee799ac: Status 404 returned error can't find the container with id 8429a488e310d4c2bffbe63d99a31569dbb572abf4c04fe1c91025c4eee799ac
Oct 30 18:38:31.237817 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:31.237788    4372 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:38:31.777501 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:31.777456    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs" event={"ID":"a9f3ac74-331f-434d-8a21-a47cc078849c","Type":"ContainerStarted","Data":"8429a488e310d4c2bffbe63d99a31569dbb572abf4c04fe1c91025c4eee799ac"}
Oct 30 18:38:32.264090 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:32.264036    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0736f3ca-42d7-4f05-9f58-64664a213a0c" path="/var/lib/kubelet/pods/0736f3ca-42d7-4f05-9f58-64664a213a0c/volumes"
Oct 30 18:38:34.798153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:34.798000    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs" event={"ID":"a9f3ac74-331f-434d-8a21-a47cc078849c","Type":"ContainerStarted","Data":"94cba841db2bf57e83705759d07151c31e601aba7a9f312e134cc98330666b12"}
Oct 30 18:38:34.798665 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:34.798190    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:34.824830 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:34.824751    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs" podStartSLOduration=1.6452413799999999 podStartE2EDuration="4.82472921s" totalImagesPullingTime="3.17948783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:34.822711402 +0000 UTC m=+328.691129732" watchObservedRunningTime="2026-10-30 18:38:34.82472921 +0000 UTC m=+328.693147650"
Oct 30 18:38:45.808865 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:38:45.808808    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-849b68f5-dflvs"
Oct 30 18:38:57.702030 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:38:57.701975    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:39:12.146301 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.146241    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"]
Oct 30 18:39:12.158086 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.158054    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.163741 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.163696    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"]
Oct 30 18:39:12.337616 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.337557    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsv2s\" (UniqueName: \"kubernetes.io/projected/02723f85-0a69-43c7-8482-c2cd80959ee3-kube-api-access-tsv2s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.337832 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.337635    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.337832 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.337727    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/02723f85-0a69-43c7-8482-c2cd80959ee3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.337832 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.337766    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.337951 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.337876    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.337951 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.337923    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.337951 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.337944    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.439016 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.438902    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.439016 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.438980    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.439288 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.439187    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.439288 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.439238    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.439288 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.439275    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsv2s\" (UniqueName: \"kubernetes.io/projected/02723f85-0a69-43c7-8482-c2cd80959ee3-kube-api-access-tsv2s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.439415 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.439347    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.439415 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.439383    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/02723f85-0a69-43c7-8482-c2cd80959ee3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.440050 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.440018    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.442083 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.442049    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.442188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.442155    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.443002 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.442951    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/02723f85-0a69-43c7-8482-c2cd80959ee3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.443113 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.443067    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.454727 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.454688    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.454883 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.454757    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsv2s\" (UniqueName: \"kubernetes.io/projected/02723f85-0a69-43c7-8482-c2cd80959ee3-kube-api-access-tsv2s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkpw4\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.474626 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.474586    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:12.765186 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.764936    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"]
Oct 30 18:39:12.768015 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:39:12.767968    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02723f85_0a69_43c7_8482_c2cd80959ee3.slice/crio-6c9774ea7724f5cab3b3e6a6ce0697b85d06e0d63f4d85d59780856c9b81389d WatchSource:0}: Error finding container 6c9774ea7724f5cab3b3e6a6ce0697b85d06e0d63f4d85d59780856c9b81389d: Status 404 returned error can't find the container with id 6c9774ea7724f5cab3b3e6a6ce0697b85d06e0d63f4d85d59780856c9b81389d
Oct 30 18:39:12.770519 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.770480    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:39:12.770627 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.770555    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:39:12.971043 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.971003    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4" event={"ID":"02723f85-0a69-43c7-8482-c2cd80959ee3","Type":"ContainerStarted","Data":"c5146c784feae6ab6a1c4ba09c9de080c7f78141ca1da7ed7a0f800bfa67c49d"}
Oct 30 18:39:12.971282 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.971067    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4" event={"ID":"02723f85-0a69-43c7-8482-c2cd80959ee3","Type":"ContainerStarted","Data":"6c9774ea7724f5cab3b3e6a6ce0697b85d06e0d63f4d85d59780856c9b81389d"}
Oct 30 18:39:12.971282 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:12.971206    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:13.977358 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:13.977319    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 18:39:14.005571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.005503    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4" podStartSLOduration=2.005485695 podStartE2EDuration="2.005485695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:13.01182092 +0000 UTC m=+366.880239250" watchObservedRunningTime="2026-10-30 18:39:14.005485695 +0000 UTC m=+367.873904024"
Oct 30 18:39:14.037096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.037057    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"]
Oct 30 18:39:14.037336 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.037312    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5" podUID="21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9" containerName="discovery" containerID="cri-o://b0e5e54fba06985154d72f57c3df2d5f0125f82605365e639b8c6922c68615e4" gracePeriod=30
Oct 30 18:39:14.304986 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.304947    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:39:14.460450 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.460389    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-csr-dns-cert\") pod \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") "
Oct 30 18:39:14.460450 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.460458    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-cacerts\" (UniqueName: \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-cacerts\") pod \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") "
Oct 30 18:39:14.460760 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.460513    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-csr-ca-configmap\") pod \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") "
Oct 30 18:39:14.460760 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.460571    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-local-certs\") pod \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") "
Oct 30 18:39:14.460760 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.460607    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-kubeconfig\") pod \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") "
Oct 30 18:39:14.460760 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.460663    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-token\" (UniqueName: \"kubernetes.io/projected/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-token\") pod \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") "
Oct 30 18:39:14.460760 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.460709    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-kube-api-access-2r6q6\" (UniqueName: \"kubernetes.io/projected/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-kube-api-access-2r6q6\") pod \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\" (UID: \"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9\") "
Oct 30 18:39:14.461103 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.461067    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-csr-ca-configmap" pod "21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9" (UID: "21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:14.464087 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.464039    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-cacerts" pod "21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9" (UID: "21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:14.464290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.464039    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-local-certs" pod "21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9" (UID: "21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:39:14.464290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.464095    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-csr-dns-cert" pod "21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9" (UID: "21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:14.464290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.464113    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-kubeconfig" pod "21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9" (UID: "21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:14.464290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.464192    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-token" pod "21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9" (UID: "21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:14.464290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.464237    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-kube-api-access-2r6q6" pod "21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9" (UID: "21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9"). InnerVolumeSpecName "kube-api-access-2r6q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:14.561966 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.561874    4372 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-csr-dns-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:14.561966 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.561909    4372 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-cacerts\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:14.561966 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.561919    4372 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-csr-ca-configmap\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:14.561966 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.561931    4372 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-local-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:14.561966 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.561939    4372 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-kubeconfig\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:14.561966 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.561947    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:14.561966 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.561956    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r6q6\" (UniqueName: \"kubernetes.io/projected/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9-kube-api-access-2r6q6\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:14.980886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.980832    4372 generic.go:356] "Generic (PLEG): container finished" podID="21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9" containerID="b0e5e54fba06985154d72f57c3df2d5f0125f82605365e639b8c6922c68615e4" exitCode=0
Oct 30 18:39:14.981431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.980915    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5" event={"ID":"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9","Type":"ContainerDied","Data":"b0e5e54fba06985154d72f57c3df2d5f0125f82605365e639b8c6922c68615e4"}
Oct 30 18:39:14.981431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.981010    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5" event={"ID":"21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9","Type":"ContainerDied","Data":"37d67c38c2af4d01fb7e4f31f8eb4958ba97a4b7035994c3e90a18ff60fb76b4"}
Oct 30 18:39:14.981431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.981041    4372 scope.go:122] "RemoveContainer" containerID="b0e5e54fba06985154d72f57c3df2d5f0125f82605365e639b8c6922c68615e4"
Oct 30 18:39:14.981431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.980921    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"
Oct 30 18:39:14.991434 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.991412    4372 scope.go:122] "RemoveContainer" containerID="b0e5e54fba06985154d72f57c3df2d5f0125f82605365e639b8c6922c68615e4"
Oct 30 18:39:14.991800 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:39:14.991779    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e5e54fba06985154d72f57c3df2d5f0125f82605365e639b8c6922c68615e4\": container with ID starting with b0e5e54fba06985154d72f57c3df2d5f0125f82605365e639b8c6922c68615e4 not found: ID does not exist" containerID="b0e5e54fba06985154d72f57c3df2d5f0125f82605365e639b8c6922c68615e4"
Oct 30 18:39:14.991849 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:14.991814    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e5e54fba06985154d72f57c3df2d5f0125f82605365e639b8c6922c68615e4"} err="failed to get container status \"b0e5e54fba06985154d72f57c3df2d5f0125f82605365e639b8c6922c68615e4\": rpc error: code = NotFound desc = could not find container \"b0e5e54fba06985154d72f57c3df2d5f0125f82605365e639b8c6922c68615e4\": container with ID starting with b0e5e54fba06985154d72f57c3df2d5f0125f82605365e639b8c6922c68615e4 not found: ID does not exist"
Oct 30 18:39:15.006168 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:15.006096    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"]
Oct 30 18:39:15.013114 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:15.013079    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l66w5"]
Oct 30 18:39:16.263533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:16.263483    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9" path="/var/lib/kubelet/pods/21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9/volumes"
Oct 30 18:39:27.698796 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:39:27.698748    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:39:34.264634 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:34.264592    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:39:45.177053 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.176997    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"]
Oct 30 18:39:45.177622 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.177361    4372 cpu_manager.go:414] "containerMap: removing container" podUID="21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9" containerName="discovery"
Oct 30 18:39:45.177622 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.177377    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9" containerName="discovery"
Oct 30 18:39:45.177622 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.177448    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="21dd3100-78ff-4e52-8af1-ad4ed7ecd9c9" containerName="discovery"
Oct 30 18:39:45.180801 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.180774    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:45.185197 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.185162    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:39:45.185401 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.185195    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:39:45.192833 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.192787    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"]
Oct 30 18:39:45.219956 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.219903    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-plgsl\""
Oct 30 18:39:45.219956 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.219961    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 18:39:45.220285 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.219960    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 18:39:45.325738 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.325684    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31-metrics-certs\") pod \"maas-api-6c665b8798-czqkv\" (UID: \"fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31\") " pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:45.325738 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.325746    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfgvk\" (UniqueName: \"kubernetes.io/projected/fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31-kube-api-access-gfgvk\") pod \"maas-api-6c665b8798-czqkv\" (UID: \"fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31\") " pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:45.325987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.325847    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31-openshift-service-ca\") pod \"maas-api-6c665b8798-czqkv\" (UID: \"fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31\") " pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:45.325987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.325939    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31-maas-api-tls\") pod \"maas-api-6c665b8798-czqkv\" (UID: \"fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31\") " pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:45.427145 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.427027    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31-metrics-certs\") pod \"maas-api-6c665b8798-czqkv\" (UID: \"fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31\") " pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:45.427145 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.427088    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgvk\" (UniqueName: \"kubernetes.io/projected/fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31-kube-api-access-gfgvk\") pod \"maas-api-6c665b8798-czqkv\" (UID: \"fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31\") " pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:45.427145 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.427118    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31-openshift-service-ca\") pod \"maas-api-6c665b8798-czqkv\" (UID: \"fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31\") " pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:45.427503 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.427174    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31-maas-api-tls\") pod \"maas-api-6c665b8798-czqkv\" (UID: \"fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31\") " pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:45.428090 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.428064    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31-openshift-service-ca\") pod \"maas-api-6c665b8798-czqkv\" (UID: \"fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31\") " pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:45.430682 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.430640    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31-metrics-certs\") pod \"maas-api-6c665b8798-czqkv\" (UID: \"fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31\") " pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:45.430866 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.430640    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31-maas-api-tls\") pod \"maas-api-6c665b8798-czqkv\" (UID: \"fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31\") " pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:45.441472 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.441402    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfgvk\" (UniqueName: \"kubernetes.io/projected/fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31-kube-api-access-gfgvk\") pod \"maas-api-6c665b8798-czqkv\" (UID: \"fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31\") " pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:45.498344 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.498291    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:45.784093 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:45.784057    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"]
Oct 30 18:39:45.786977 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:39:45.786933    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1bf3b0_ef42_4659_9d4e_5e7dbc3e1c31.slice/crio-61ab843a298c739e8d08d6e965ea064e2fd8f3deaf21fe52dbea3b1d88ccd8ed WatchSource:0}: Error finding container 61ab843a298c739e8d08d6e965ea064e2fd8f3deaf21fe52dbea3b1d88ccd8ed: Status 404 returned error can't find the container with id 61ab843a298c739e8d08d6e965ea064e2fd8f3deaf21fe52dbea3b1d88ccd8ed
Oct 30 18:39:46.129756 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:46.129710    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv" event={"ID":"fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31","Type":"ContainerStarted","Data":"61ab843a298c739e8d08d6e965ea064e2fd8f3deaf21fe52dbea3b1d88ccd8ed"}
Oct 30 18:39:49.146380 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:49.146340    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv" event={"ID":"fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31","Type":"ContainerStarted","Data":"87f54f680d21e277295fcbe8c812a3d6eb4d9f4258984570cea87a03cb645cf2"}
Oct 30 18:39:49.146859 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:49.146639    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:49.170106 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:49.170032    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv" podStartSLOduration=1.758330835 podStartE2EDuration="4.170011979s" totalImagesPullingTime="2.411681144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:49.168728376 +0000 UTC m=+403.037146707" watchObservedRunningTime="2026-10-30 18:39:49.170011979 +0000 UTC m=+403.038430309"
Oct 30 18:39:55.158058 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:39:55.158014    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c665b8798-czqkv"
Oct 30 18:39:57.699459 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:39:57.699402    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:40:27.700494 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:40:27.700441    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:40:57.699345 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:40:57.699293    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:41:06.954810 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:06.954765    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2"]
Oct 30 18:41:06.958532 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:06.958502    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2"
Oct 30 18:41:06.967345 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:06.967320    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 18:41:06.971272 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:06.971232    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2"]
Oct 30 18:41:07.045961 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:07.045874    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvtlr\" (UniqueName: \"kubernetes.io/projected/ac73b3bf-215f-43f2-afa9-b5155af04781-kube-api-access-nvtlr\") pod \"payload-pre-processing-7b98bb678c-m4bn2\" (UID: \"ac73b3bf-215f-43f2-afa9-b5155af04781\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2"
Oct 30 18:41:07.046256 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:07.046046    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac73b3bf-215f-43f2-afa9-b5155af04781-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-m4bn2\" (UID: \"ac73b3bf-215f-43f2-afa9-b5155af04781\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2"
Oct 30 18:41:07.147388 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:07.147335    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac73b3bf-215f-43f2-afa9-b5155af04781-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-m4bn2\" (UID: \"ac73b3bf-215f-43f2-afa9-b5155af04781\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2"
Oct 30 18:41:07.147686 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:07.147419    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvtlr\" (UniqueName: \"kubernetes.io/projected/ac73b3bf-215f-43f2-afa9-b5155af04781-kube-api-access-nvtlr\") pod \"payload-pre-processing-7b98bb678c-m4bn2\" (UID: \"ac73b3bf-215f-43f2-afa9-b5155af04781\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2"
Oct 30 18:41:07.148594 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:07.148559    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac73b3bf-215f-43f2-afa9-b5155af04781-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-m4bn2\" (UID: \"ac73b3bf-215f-43f2-afa9-b5155af04781\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2"
Oct 30 18:41:07.161211 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:07.161173    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvtlr\" (UniqueName: \"kubernetes.io/projected/ac73b3bf-215f-43f2-afa9-b5155af04781-kube-api-access-nvtlr\") pod \"payload-pre-processing-7b98bb678c-m4bn2\" (UID: \"ac73b3bf-215f-43f2-afa9-b5155af04781\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2"
Oct 30 18:41:07.274105 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:07.273972    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2"
Oct 30 18:41:07.558530 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:07.558495    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2"]
Oct 30 18:41:07.561818 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:41:07.561773    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac73b3bf_215f_43f2_afa9_b5155af04781.slice/crio-d206b3c26f65ac7c3a8f3ec1c059e98a934351bf5f18adf5b393a03c6c9bb4fa WatchSource:0}: Error finding container d206b3c26f65ac7c3a8f3ec1c059e98a934351bf5f18adf5b393a03c6c9bb4fa: Status 404 returned error can't find the container with id d206b3c26f65ac7c3a8f3ec1c059e98a934351bf5f18adf5b393a03c6c9bb4fa
Oct 30 18:41:08.504492 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:08.504440    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2" event={"ID":"ac73b3bf-215f-43f2-afa9-b5155af04781","Type":"ContainerStarted","Data":"d206b3c26f65ac7c3a8f3ec1c059e98a934351bf5f18adf5b393a03c6c9bb4fa"}
Oct 30 18:41:11.520424 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:11.520375    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2" event={"ID":"ac73b3bf-215f-43f2-afa9-b5155af04781","Type":"ContainerStarted","Data":"9bdf4dec2f320d7cbb48da3b7540ee4860d67ee95040bad865a78abe17cb316d"}
Oct 30 18:41:11.520947 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:11.520566    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2"
Oct 30 18:41:11.542255 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:11.542173    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2" podStartSLOduration=2.646494266 podStartE2EDuration="5.542153867s" totalImagesPullingTime="2.895659601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:11.540316758 +0000 UTC m=+485.408735089" watchObservedRunningTime="2026-10-30 18:41:11.542153867 +0000 UTC m=+485.410572194"
Oct 30 18:41:22.528579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:22.528483    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-m4bn2"
Oct 30 18:41:27.700431 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:41:27.700380    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:41:38.100411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.100362    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"]
Oct 30 18:41:38.104275 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.104241    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.108794 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.108746    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 18:41:38.108794 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.108801    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 18:41:38.109936 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.109910    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zfsfz\""
Oct 30 18:41:38.109936 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.109927    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 18:41:38.115519 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.115478    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"]
Oct 30 18:41:38.238292 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.238244    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnpr\" (UniqueName: \"kubernetes.io/projected/74a03d36-6248-4cfd-be2f-9415834a3337-kube-api-access-4dnpr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.238292 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.238291    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74a03d36-6248-4cfd-be2f-9415834a3337-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.238578 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.238327    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74a03d36-6248-4cfd-be2f-9415834a3337-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.238578 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.238495    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a03d36-6248-4cfd-be2f-9415834a3337-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.238578 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.238526    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a03d36-6248-4cfd-be2f-9415834a3337-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.238578 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.238556    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74a03d36-6248-4cfd-be2f-9415834a3337-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.339597 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.339548    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74a03d36-6248-4cfd-be2f-9415834a3337-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.339883 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.339658    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a03d36-6248-4cfd-be2f-9415834a3337-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.339883 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.339680    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a03d36-6248-4cfd-be2f-9415834a3337-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.339883 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.339707    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74a03d36-6248-4cfd-be2f-9415834a3337-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.339883 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.339770    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnpr\" (UniqueName: \"kubernetes.io/projected/74a03d36-6248-4cfd-be2f-9415834a3337-kube-api-access-4dnpr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.339883 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.339801    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74a03d36-6248-4cfd-be2f-9415834a3337-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.340249 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.340204    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a03d36-6248-4cfd-be2f-9415834a3337-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.340424 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.340334    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74a03d36-6248-4cfd-be2f-9415834a3337-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.340424 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.340347    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74a03d36-6248-4cfd-be2f-9415834a3337-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.342714 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.342670    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74a03d36-6248-4cfd-be2f-9415834a3337-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.342983 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.342961    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a03d36-6248-4cfd-be2f-9415834a3337-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.354563 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.354472    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnpr\" (UniqueName: \"kubernetes.io/projected/74a03d36-6248-4cfd-be2f-9415834a3337-kube-api-access-4dnpr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp\" (UID: \"74a03d36-6248-4cfd-be2f-9415834a3337\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.419216 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.419115    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:38.582138 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.582076    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:41:38.707544 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:38.707504    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"]
Oct 30 18:41:38.710249 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:41:38.710215    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a03d36_6248_4cfd_be2f_9415834a3337.slice/crio-eb6b1faf6d8597cbc9734460739697309a8b46b16499ee5997378d2e6afa5855 WatchSource:0}: Error finding container eb6b1faf6d8597cbc9734460739697309a8b46b16499ee5997378d2e6afa5855: Status 404 returned error can't find the container with id eb6b1faf6d8597cbc9734460739697309a8b46b16499ee5997378d2e6afa5855
Oct 30 18:41:39.649575 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:39.649528    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp" event={"ID":"74a03d36-6248-4cfd-be2f-9415834a3337","Type":"ContainerStarted","Data":"eb6b1faf6d8597cbc9734460739697309a8b46b16499ee5997378d2e6afa5855"}
Oct 30 18:41:40.545664 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:40.545528    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:41:40.656650 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:40.656613    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp" event={"ID":"74a03d36-6248-4cfd-be2f-9415834a3337","Type":"ContainerStarted","Data":"0a09d3d75ef95022e9faa5072817e0a99037b24b0a1134d1e3fa23a4ef5ae5d4"}
Oct 30 18:41:40.681537 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:40.681473    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp" podStartSLOduration=1.298416145 podStartE2EDuration="2.681454918s" totalImagesPullingTime="1.383038773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:40.680396766 +0000 UTC m=+514.548815087" watchObservedRunningTime="2026-10-30 18:41:40.681454918 +0000 UTC m=+514.549873248"
Oct 30 18:41:41.419336 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:41.419288    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:41.910298 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:41.910242    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"]
Oct 30 18:41:41.913992 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:41.913966    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:41.920503 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:41.920468    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 18:41:41.925518 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:41.925483    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"]
Oct 30 18:41:42.077561 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.077506    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b82fd5-5bcc-40ae-b071-4e026abe1d07-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.077561 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.077561    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b82fd5-5bcc-40ae-b071-4e026abe1d07-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.077826 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.077652    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvb7q\" (UniqueName: \"kubernetes.io/projected/45b82fd5-5bcc-40ae-b071-4e026abe1d07-kube-api-access-nvb7q\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.077826 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.077706    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b82fd5-5bcc-40ae-b071-4e026abe1d07-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.077826 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.077766    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b82fd5-5bcc-40ae-b071-4e026abe1d07-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.077826 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.077814    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b82fd5-5bcc-40ae-b071-4e026abe1d07-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.178995 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.178831    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b82fd5-5bcc-40ae-b071-4e026abe1d07-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.178995 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.178895    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b82fd5-5bcc-40ae-b071-4e026abe1d07-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.178995 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.178952    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b82fd5-5bcc-40ae-b071-4e026abe1d07-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.178995 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.178980    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b82fd5-5bcc-40ae-b071-4e026abe1d07-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.178995 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.179004    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b82fd5-5bcc-40ae-b071-4e026abe1d07-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.179527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.179225    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvb7q\" (UniqueName: \"kubernetes.io/projected/45b82fd5-5bcc-40ae-b071-4e026abe1d07-kube-api-access-nvb7q\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.179527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.179486    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b82fd5-5bcc-40ae-b071-4e026abe1d07-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.179641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.179530    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b82fd5-5bcc-40ae-b071-4e026abe1d07-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.179641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.179564    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b82fd5-5bcc-40ae-b071-4e026abe1d07-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.181970 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.181944    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b82fd5-5bcc-40ae-b071-4e026abe1d07-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.182323 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.182298    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b82fd5-5bcc-40ae-b071-4e026abe1d07-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.194723 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.194672    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvb7q\" (UniqueName: \"kubernetes.io/projected/45b82fd5-5bcc-40ae-b071-4e026abe1d07-kube-api-access-nvb7q\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jn2fc\" (UID: \"45b82fd5-5bcc-40ae-b071-4e026abe1d07\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.232736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.232681    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:42.521424 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.521306    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"]
Oct 30 18:41:42.524551 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:41:42.524503    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b82fd5_5bcc_40ae_b071_4e026abe1d07.slice/crio-1a8dbcc91d7e0cfa53dc8de5764a095d8b9382cac4ff24e3ee135d635f3190e0 WatchSource:0}: Error finding container 1a8dbcc91d7e0cfa53dc8de5764a095d8b9382cac4ff24e3ee135d635f3190e0: Status 404 returned error can't find the container with id 1a8dbcc91d7e0cfa53dc8de5764a095d8b9382cac4ff24e3ee135d635f3190e0
Oct 30 18:41:42.667614 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:42.667560    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc" event={"ID":"45b82fd5-5bcc-40ae-b071-4e026abe1d07","Type":"ContainerStarted","Data":"1a8dbcc91d7e0cfa53dc8de5764a095d8b9382cac4ff24e3ee135d635f3190e0"}
Oct 30 18:41:43.382899 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:43.382864    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:41:43.674974 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:43.674845    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc" event={"ID":"45b82fd5-5bcc-40ae-b071-4e026abe1d07","Type":"ContainerStarted","Data":"b301c2dc2b31b95b9bd97d3ea25c3c4efd468319a02867ef2dfa96d839d075f0"}
Oct 30 18:41:43.698788 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:43.698724    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc" podStartSLOduration=2.383760374 podStartE2EDuration="2.698707572s" totalImagesPullingTime="314.947198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:43.696901903 +0000 UTC m=+517.565320261" watchObservedRunningTime="2026-10-30 18:41:43.698707572 +0000 UTC m=+517.567125902"
Oct 30 18:41:44.233887 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:44.233839    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:45.744248 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:45.744196    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-9mxbq"]
Oct 30 18:41:45.751770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:45.751729    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:45.773050 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:45.773002    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-9mxbq"]
Oct 30 18:41:45.776492 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:45.776458    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 18:41:45.856304 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:45.856245    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:41:45.915857 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:45.915801    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0316c797-de93-4782-9506-7057fb3aa4a1-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:45.915857 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:45.915859    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqh4x\" (UniqueName: \"kubernetes.io/projected/0316c797-de93-4782-9506-7057fb3aa4a1-kube-api-access-lqh4x\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:45.916175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:45.915991    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0316c797-de93-4782-9506-7057fb3aa4a1-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:45.916175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:45.916027    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0316c797-de93-4782-9506-7057fb3aa4a1-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:45.916175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:45.916058    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0316c797-de93-4782-9506-7057fb3aa4a1-home\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:45.916339 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:45.916178    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0316c797-de93-4782-9506-7057fb3aa4a1-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:46.017261 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:46.017096    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0316c797-de93-4782-9506-7057fb3aa4a1-home\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:46.017541 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:46.017305    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0316c797-de93-4782-9506-7057fb3aa4a1-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:46.017541 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:46.017391    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0316c797-de93-4782-9506-7057fb3aa4a1-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:46.017541 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:46.017474    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqh4x\" (UniqueName: \"kubernetes.io/projected/0316c797-de93-4782-9506-7057fb3aa4a1-kube-api-access-lqh4x\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:46.017541 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:46.017524    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0316c797-de93-4782-9506-7057fb3aa4a1-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:46.017791 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:46.017564    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0316c797-de93-4782-9506-7057fb3aa4a1-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:46.017791 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:46.017690    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0316c797-de93-4782-9506-7057fb3aa4a1-home\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:46.017791 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:46.017751    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0316c797-de93-4782-9506-7057fb3aa4a1-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:46.017962 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:46.017936    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0316c797-de93-4782-9506-7057fb3aa4a1-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:46.020343 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:46.020303    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0316c797-de93-4782-9506-7057fb3aa4a1-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:46.020498 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:46.020446    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0316c797-de93-4782-9506-7057fb3aa4a1-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:46.041876 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:46.041824    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqh4x\" (UniqueName: \"kubernetes.io/projected/0316c797-de93-4782-9506-7057fb3aa4a1-kube-api-access-lqh4x\") pod \"echo-simulator-kserve-64b6f8f96c-9mxbq\" (UID: \"0316c797-de93-4782-9506-7057fb3aa4a1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:46.066679 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:46.066619    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:46.368608 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:46.368573    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-9mxbq"]
Oct 30 18:41:46.371462 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:41:46.371424    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0316c797_de93_4782_9506_7057fb3aa4a1.slice/crio-1c350e2b2d0a0dbfcad166489b1823fcec52c7859f8b90e993fd64aed5c1dd0a WatchSource:0}: Error finding container 1c350e2b2d0a0dbfcad166489b1823fcec52c7859f8b90e993fd64aed5c1dd0a: Status 404 returned error can't find the container with id 1c350e2b2d0a0dbfcad166489b1823fcec52c7859f8b90e993fd64aed5c1dd0a
Oct 30 18:41:46.690629 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:46.690530    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq" event={"ID":"0316c797-de93-4782-9506-7057fb3aa4a1","Type":"ContainerStarted","Data":"1c350e2b2d0a0dbfcad166489b1823fcec52c7859f8b90e993fd64aed5c1dd0a"}
Oct 30 18:41:47.826625 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:47.826571    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"]
Oct 30 18:41:47.830569 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:47.830533    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:47.838246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:47.838213    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 18:41:47.847391 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:47.847341    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"]
Oct 30 18:41:47.935272 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:47.935221    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7aab0a-df8b-486f-a479-bab120d33d03-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:47.935486 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:47.935280    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7aab0a-df8b-486f-a479-bab120d33d03-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:47.935486 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:47.935311    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7aab0a-df8b-486f-a479-bab120d33d03-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:47.935486 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:47.935389    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7aab0a-df8b-486f-a479-bab120d33d03-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:47.935628 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:47.935546    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5k4\" (UniqueName: \"kubernetes.io/projected/0a7aab0a-df8b-486f-a479-bab120d33d03-kube-api-access-9z5k4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:47.935628 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:47.935595    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7aab0a-df8b-486f-a479-bab120d33d03-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:48.037204 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.037149    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7aab0a-df8b-486f-a479-bab120d33d03-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:48.037204 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.037211    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7aab0a-df8b-486f-a479-bab120d33d03-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:48.037520 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.037235    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7aab0a-df8b-486f-a479-bab120d33d03-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:48.037520 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.037264    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7aab0a-df8b-486f-a479-bab120d33d03-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:48.037520 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.037313    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5k4\" (UniqueName: \"kubernetes.io/projected/0a7aab0a-df8b-486f-a479-bab120d33d03-kube-api-access-9z5k4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:48.037520 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.037341    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7aab0a-df8b-486f-a479-bab120d33d03-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:48.037903 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.037698    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7aab0a-df8b-486f-a479-bab120d33d03-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:48.037903 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.037876    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7aab0a-df8b-486f-a479-bab120d33d03-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:48.038112 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.037883    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7aab0a-df8b-486f-a479-bab120d33d03-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:48.040208 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.040180    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7aab0a-df8b-486f-a479-bab120d33d03-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:48.040695 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.040664    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7aab0a-df8b-486f-a479-bab120d33d03-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:48.056463 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.056413    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5k4\" (UniqueName: \"kubernetes.io/projected/0a7aab0a-df8b-486f-a479-bab120d33d03-kube-api-access-9z5k4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm\" (UID: \"0a7aab0a-df8b-486f-a479-bab120d33d03\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:48.150740 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.150643    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:48.419744 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.419648    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:48.433774 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.433740    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:48.439616 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.439577    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"]
Oct 30 18:41:48.442557 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:41:48.442526    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7aab0a_df8b_486f_a479_bab120d33d03.slice/crio-d04dd4d32a19f0edf85efa1eb5a491271ef25ca8b0be1267f478d2a2a68a4b26 WatchSource:0}: Error finding container d04dd4d32a19f0edf85efa1eb5a491271ef25ca8b0be1267f478d2a2a68a4b26: Status 404 returned error can't find the container with id d04dd4d32a19f0edf85efa1eb5a491271ef25ca8b0be1267f478d2a2a68a4b26
Oct 30 18:41:48.703482 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.703360    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq" event={"ID":"0316c797-de93-4782-9506-7057fb3aa4a1","Type":"ContainerStarted","Data":"4c2e447bd31a7365a898cfce4f30fe5bb1ebd90f98f506343b15e35f8b846332"}
Oct 30 18:41:48.712418 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.712360    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm" event={"ID":"0a7aab0a-df8b-486f-a479-bab120d33d03","Type":"ContainerStarted","Data":"d04dd4d32a19f0edf85efa1eb5a491271ef25ca8b0be1267f478d2a2a68a4b26"}
Oct 30 18:41:48.725567 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.725531    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp"
Oct 30 18:41:48.727307 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:48.727242    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq" podStartSLOduration=1.89454452 podStartE2EDuration="3.727225149s" totalImagesPullingTime="1.832680629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:48.726381638 +0000 UTC m=+522.594799968" watchObservedRunningTime="2026-10-30 18:41:48.727225149 +0000 UTC m=+522.595643481"
Oct 30 18:41:49.066840 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:49.066744    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:49.719320 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:49.719260    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm" event={"ID":"0a7aab0a-df8b-486f-a479-bab120d33d03","Type":"ContainerStarted","Data":"edf46a41fd7d7276bb0a059ebf37319394565d2d4dc2590d89ab6c0b4e0fc863"}
Oct 30 18:41:49.763975 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:49.763901    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm" podStartSLOduration=2.4185180490000002 podStartE2EDuration="2.763879728s" totalImagesPullingTime="345.361679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:49.763823679 +0000 UTC m=+523.632242012" watchObservedRunningTime="2026-10-30 18:41:49.763879728 +0000 UTC m=+523.632298106"
Oct 30 18:41:49.891061 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:49.891014    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:41:50.151159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:50.151062    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:51.081203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.081157    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"]
Oct 30 18:41:51.084861 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.084838    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.089689 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.089655    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-n2l29\""
Oct 30 18:41:51.107971 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.107931    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"]
Oct 30 18:41:51.168273 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.168189    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.168855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.168340    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.168855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.168392    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.168855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.168412    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.168855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.168520    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhc5g\" (UniqueName: \"kubernetes.io/projected/a588c60a-b15e-4d10-a2cc-71eb915006db-kube-api-access-vhc5g\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.168855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.168585    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a588c60a-b15e-4d10-a2cc-71eb915006db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.168855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.168637    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.168855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.168668    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.168855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.168706    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.269450 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.269394    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhc5g\" (UniqueName: \"kubernetes.io/projected/a588c60a-b15e-4d10-a2cc-71eb915006db-kube-api-access-vhc5g\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.269731 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.269501    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a588c60a-b15e-4d10-a2cc-71eb915006db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.269731 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.269535    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.269731 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.269562    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.269731 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.269611    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.269731 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.269642    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.269731 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.269697    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.269994 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.269757    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.269994 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.269783    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.270113 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.270083    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.270210 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.270188    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.270270 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.270250    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.270352 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.270329    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.270438 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.270418    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a588c60a-b15e-4d10-a2cc-71eb915006db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.272767 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.272718    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.272954 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.272934    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.283700 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.283652    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhc5g\" (UniqueName: \"kubernetes.io/projected/a588c60a-b15e-4d10-a2cc-71eb915006db-kube-api-access-vhc5g\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.284051 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.284029    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-fl7cg\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.400543 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.400410    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:51.690748 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.690714    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"]
Oct 30 18:41:51.693579 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:41:51.693546    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda588c60a_b15e_4d10_a2cc_71eb915006db.slice/crio-9cbc573dba2805e62b104a74078c14cf1fcfa74c64638c7229c7c64c38eabe3f WatchSource:0}: Error finding container 9cbc573dba2805e62b104a74078c14cf1fcfa74c64638c7229c7c64c38eabe3f: Status 404 returned error can't find the container with id 9cbc573dba2805e62b104a74078c14cf1fcfa74c64638c7229c7c64c38eabe3f
Oct 30 18:41:51.731353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:51.731285    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg" event={"ID":"a588c60a-b15e-4d10-a2cc-71eb915006db","Type":"ContainerStarted","Data":"9cbc573dba2805e62b104a74078c14cf1fcfa74c64638c7229c7c64c38eabe3f"}
Oct 30 18:41:52.233328 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:52.233254    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:52.246662 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:52.246619    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:52.748505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:52.748469    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jn2fc"
Oct 30 18:41:54.910977 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:54.910929    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:41:54.911642 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:54.911031    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:41:54.911642 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:54.911072    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:41:55.755592 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:55.755541    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg" event={"ID":"a588c60a-b15e-4d10-a2cc-71eb915006db","Type":"ContainerStarted","Data":"6a3302c9cbd0f68b7b35a214174641da9162469b7d91f92539ac8d1a8ebf1db4"}
Oct 30 18:41:55.805668 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:55.805601    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg" podStartSLOduration=1.591643178 podStartE2EDuration="4.805578401s" totalImagesPullingTime="3.213935223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:55.803741907 +0000 UTC m=+529.672160240" watchObservedRunningTime="2026-10-30 18:41:55.805578401 +0000 UTC m=+529.673996730"
Oct 30 18:41:56.067600 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:56.067472    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:56.103193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:56.099697    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:56.403422 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:56.403370    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:41:56.405291 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:56.405254    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fl7cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 18:41:56.405504 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:56.405353    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg" podUID="a588c60a-b15e-4d10-a2cc-71eb915006db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 18:41:56.772743 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:56.772645    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq"
Oct 30 18:41:57.403738 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:57.403685    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fl7cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 18:41:57.404365 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:57.403768    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg" podUID="a588c60a-b15e-4d10-a2cc-71eb915006db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 18:41:57.720769 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:41:57.720649    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:41:58.151303 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:58.151229    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:58.165378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:58.165342    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:58.403832 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:58.403717    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fl7cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 18:41:58.403832 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:58.403793    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg" podUID="a588c60a-b15e-4d10-a2cc-71eb915006db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 18:41:58.803617 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:58.803470    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm"
Oct 30 18:41:59.403777 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:59.403715    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fl7cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 18:41:59.403999 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:41:59.403819    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg" podUID="a588c60a-b15e-4d10-a2cc-71eb915006db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 18:42:00.402162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:00.401292    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fl7cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 18:42:00.402162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:00.401380    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg" podUID="a588c60a-b15e-4d10-a2cc-71eb915006db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 18:42:01.401090 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:01.401045    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fl7cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 18:42:01.401624 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:01.401112    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg" podUID="a588c60a-b15e-4d10-a2cc-71eb915006db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 18:42:02.401818 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:02.401767    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-fl7cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 18:42:02.402366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:02.401848    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg" podUID="a588c60a-b15e-4d10-a2cc-71eb915006db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 18:42:03.767246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:03.767169    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:42:03.767792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:03.767607    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:42:03.768641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:03.768620    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 18:42:21.208392 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.208346    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"]
Oct 30 18:42:21.217064 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.217016    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.243495 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.243450    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"]
Oct 30 18:42:21.263460 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.263413    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"]
Oct 30 18:42:21.268332 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.268297    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.291355 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.291309    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"]
Oct 30 18:42:21.334383 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.334329    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1f4775-99b1-4896-b58f-348c46e3b5f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.334660 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.334410    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.334660 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.334524    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.334660 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.334578    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1f4775-99b1-4896-b58f-348c46e3b5f4-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.334660 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.334616    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1f4775-99b1-4896-b58f-348c46e3b5f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.334988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.334797    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.334988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.334862    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1f4775-99b1-4896-b58f-348c46e3b5f4-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.334988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.334888    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1f4775-99b1-4896-b58f-348c46e3b5f4-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.334988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.334948    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5vn\" (UniqueName: \"kubernetes.io/projected/ce1f4775-99b1-4896-b58f-348c46e3b5f4-kube-api-access-4h5vn\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.334988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.334977    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.335365 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.335018    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.335365 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.335095    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f4775-99b1-4896-b58f-348c46e3b5f4-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.335365 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.335167    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f4775-99b1-4896-b58f-348c46e3b5f4-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.335365 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.335261    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.335365 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.335309    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vklkz\" (UniqueName: \"kubernetes.io/projected/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-kube-api-access-vklkz\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.335619 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.335374    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.335619 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.335406    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1f4775-99b1-4896-b58f-348c46e3b5f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.335720 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.335665    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.436323 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.436265    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f4775-99b1-4896-b58f-348c46e3b5f4-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.436586 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.436339    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.436586 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.436376    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vklkz\" (UniqueName: \"kubernetes.io/projected/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-kube-api-access-vklkz\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.436586 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.436410    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.436586 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.436440    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1f4775-99b1-4896-b58f-348c46e3b5f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.436881 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.436587    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.436881 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.436662    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1f4775-99b1-4896-b58f-348c46e3b5f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.436881 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.436725    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.436881 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.436763    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.436881 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.436817    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1f4775-99b1-4896-b58f-348c46e3b5f4-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.436881 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.436838    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f4775-99b1-4896-b58f-348c46e3b5f4-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.437237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.436841    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1f4775-99b1-4896-b58f-348c46e3b5f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.437237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.436937    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.437237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.436972    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1f4775-99b1-4896-b58f-348c46e3b5f4-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.437237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.436980    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.437237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.437006    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1f4775-99b1-4896-b58f-348c46e3b5f4-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.437237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.437051    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5vn\" (UniqueName: \"kubernetes.io/projected/ce1f4775-99b1-4896-b58f-348c46e3b5f4-kube-api-access-4h5vn\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.437237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.437086    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.437237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.437172    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.437237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.437235    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.437698 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.437270    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f4775-99b1-4896-b58f-348c46e3b5f4-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.437698 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.437396    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.437698 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.437676    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1f4775-99b1-4896-b58f-348c46e3b5f4-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.437874 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.437700    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.437874 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.437730    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.437874 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.437770    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1f4775-99b1-4896-b58f-348c46e3b5f4-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.438080 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.438052    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1f4775-99b1-4896-b58f-348c46e3b5f4-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.438273 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.438251    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1f4775-99b1-4896-b58f-348c46e3b5f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.439672 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.439633    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1f4775-99b1-4896-b58f-348c46e3b5f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.439823 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.439702    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.440630 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.440600    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.440744 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.440640    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1f4775-99b1-4896-b58f-348c46e3b5f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.459654 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.459519    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1f4775-99b1-4896-b58f-348c46e3b5f4-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.459654 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.459602    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.459654 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.459625    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vklkz\" (UniqueName: \"kubernetes.io/projected/ccc2f38a-84e5-43c3-98d3-0f39ea658a2e-kube-api-access-vklkz\") pod \"maas-default-gateway-openshift-default-855f9574cf-998xh\" (UID: \"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.459654 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.459628    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5vn\" (UniqueName: \"kubernetes.io/projected/ce1f4775-99b1-4896-b58f-348c46e3b5f4-kube-api-access-4h5vn\") pod \"maas-default-gateway-openshift-default-855f9574cf-qstgx\" (UID: \"ce1f4775-99b1-4896-b58f-348c46e3b5f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.534219 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.534116    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:21.583919 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.583864    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:21.840384 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.838229    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"]
Oct 30 18:42:21.846460 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.846402    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:21.846612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.846511    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:21.846612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.846556    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:21.887211 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.887177    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"]
Oct 30 18:42:21.896651 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:42:21.896608    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f4775_99b1_4896_b58f_348c46e3b5f4.slice/crio-11a983dba64258c907f248600cb86d1fa653ec60bad07107514218d155caf022 WatchSource:0}: Error finding container 11a983dba64258c907f248600cb86d1fa653ec60bad07107514218d155caf022: Status 404 returned error can't find the container with id 11a983dba64258c907f248600cb86d1fa653ec60bad07107514218d155caf022
Oct 30 18:42:21.899528 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.899482    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:21.899754 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.899577    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:21.899754 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.899622    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:21.905466 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.905424    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh" event={"ID":"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e","Type":"ContainerStarted","Data":"e48161226fa0b2a606c51988ae6661e69112cb6c3fd0891aa491da116ce965b6"}
Oct 30 18:42:21.907138 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:21.907088    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx" event={"ID":"ce1f4775-99b1-4896-b58f-348c46e3b5f4","Type":"ContainerStarted","Data":"11a983dba64258c907f248600cb86d1fa653ec60bad07107514218d155caf022"}
Oct 30 18:42:22.929094 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:22.926270    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx" event={"ID":"ce1f4775-99b1-4896-b58f-348c46e3b5f4","Type":"ContainerStarted","Data":"cb3c90fa26141dbfd0f202e6ce8e21a86c94386b36f944f02455098f68bedb9e"}
Oct 30 18:42:22.932190 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:22.932117    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh" event={"ID":"ccc2f38a-84e5-43c3-98d3-0f39ea658a2e","Type":"ContainerStarted","Data":"b86a71762e8a0e2acdfd1660c21f437ac3d3d96d1a157694c18a6f6b31082127"}
Oct 30 18:42:22.957752 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:22.957681    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx" podStartSLOduration=1.95765975 podStartE2EDuration="1.95765975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:22.954821331 +0000 UTC m=+556.823239661" watchObservedRunningTime="2026-10-30 18:42:22.95765975 +0000 UTC m=+556.826078194"
Oct 30 18:42:23.535205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:23.535111    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:23.537210 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:23.537176    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-998xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 18:42:23.537392 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:23.537233    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh" podUID="ccc2f38a-84e5-43c3-98d3-0f39ea658a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 18:42:23.586689 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:23.586637    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:23.611193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:23.592344    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qstgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 18:42:23.611193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:23.592442    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx" podUID="ce1f4775-99b1-4896-b58f-348c46e3b5f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 18:42:24.534800 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:24.534738    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-998xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 18:42:24.535335 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:24.534833    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh" podUID="ccc2f38a-84e5-43c3-98d3-0f39ea658a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 18:42:24.585787 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:24.585734    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qstgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 18:42:24.586205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:24.586161    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx" podUID="ce1f4775-99b1-4896-b58f-348c46e3b5f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 18:42:25.534727 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:25.534664    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-998xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 18:42:25.534972 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:25.534741    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh" podUID="ccc2f38a-84e5-43c3-98d3-0f39ea658a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 18:42:25.589170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:25.588833    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qstgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 18:42:25.589170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:25.588910    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx" podUID="ce1f4775-99b1-4896-b58f-348c46e3b5f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 18:42:26.534993 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:26.534930    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-998xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 18:42:26.535521 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:26.535015    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh" podUID="ccc2f38a-84e5-43c3-98d3-0f39ea658a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 18:42:26.587749 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:26.587692    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qstgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 18:42:26.588175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:26.588101    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx" podUID="ce1f4775-99b1-4896-b58f-348c46e3b5f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 18:42:27.535735 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:27.535683    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-998xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 18:42:27.536255 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:27.535771    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh" podUID="ccc2f38a-84e5-43c3-98d3-0f39ea658a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 18:42:27.590169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:27.586720    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qstgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 18:42:27.590169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:27.586810    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx" podUID="ce1f4775-99b1-4896-b58f-348c46e3b5f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 18:42:27.747440 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:42:27.747323    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:42:28.535535 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:28.535473    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-998xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 18:42:28.535779 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:28.535569    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh" podUID="ccc2f38a-84e5-43c3-98d3-0f39ea658a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 18:42:28.585456 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:28.585395    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qstgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 18:42:28.585683 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:28.585478    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx" podUID="ce1f4775-99b1-4896-b58f-348c46e3b5f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 18:42:29.534851 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:29.534809    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-998xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 18:42:29.535077 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:29.534876    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh" podUID="ccc2f38a-84e5-43c3-98d3-0f39ea658a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 18:42:29.585105 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:29.585051    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qstgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 18:42:29.585697 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:29.585162    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx" podUID="ce1f4775-99b1-4896-b58f-348c46e3b5f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 18:42:30.901019 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:30.900970    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:30.902518 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:30.902491    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:30.903652 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:30.903615    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh"
Oct 30 18:42:30.934424 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:30.934344    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh" podStartSLOduration=9.934325779 podStartE2EDuration="9.934325779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:23.000268937 +0000 UTC m=+556.868687288" watchObservedRunningTime="2026-10-30 18:42:30.934325779 +0000 UTC m=+564.802744109"
Oct 30 18:42:31.121681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:31.121641    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:31.122217 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:31.122180    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:31.123179 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:31.123151    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx"
Oct 30 18:42:35.749370 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.749320    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"]
Oct 30 18:42:35.753526 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.753487    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.777168 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.777059    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-78934afe-gw-openshift-default-dockercfg-w46pz\""
Oct 30 18:42:35.781011 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.780964    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"]
Oct 30 18:42:35.843568 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.843503    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-data\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.843568 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.843562    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-podinfo\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.843887 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.843662    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-credential-socket\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.843887 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.843703    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-envoy\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.843887 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.843767    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlpxd\" (UniqueName: \"kubernetes.io/projected/2b704d30-26f4-4f7f-9b78-aece8de84de0-kube-api-access-nlpxd\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.844069 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.843896    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-workload-socket\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.844069 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.843932    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b704d30-26f4-4f7f-9b78-aece8de84de0-istiod-ca-cert\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.844069 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.844020    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-token\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.844234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.844104    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-workload-certs\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.945495 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.945447    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-envoy\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.945495 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.945496    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlpxd\" (UniqueName: \"kubernetes.io/projected/2b704d30-26f4-4f7f-9b78-aece8de84de0-kube-api-access-nlpxd\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.945818 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.945554    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-workload-socket\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.945818 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.945571    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b704d30-26f4-4f7f-9b78-aece8de84de0-istiod-ca-cert\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.945818 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.945631    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-token\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.945818 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.945667    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-workload-certs\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.945818 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.945744    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-data\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.945818 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.945793    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-podinfo\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.946099 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.945954    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-credential-socket\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.946099 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.946081    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-workload-certs\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.946221 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.946113    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-data\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.946401 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.946377    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-credential-socket\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.946471 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.946437    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-workload-socket\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.946862 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.946825    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b704d30-26f4-4f7f-9b78-aece8de84de0-istiod-ca-cert\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.949513 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.949474    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-envoy\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.949666 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.949606    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-podinfo\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.960774 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.960739    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-token\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:35.960970 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:35.960949    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlpxd\" (UniqueName: \"kubernetes.io/projected/2b704d30-26f4-4f7f-9b78-aece8de84de0-kube-api-access-nlpxd\") pod \"e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") " pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:36.071972 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:36.071869    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:36.393221 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:36.393109    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"]
Oct 30 18:42:36.394217 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:42:36.394168    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b704d30_26f4_4f7f_9b78_aece8de84de0.slice/crio-0d3dcddd3b2e4e39dba90039c8f44372cff5f483a369122ce0aa0adfb37daf6a WatchSource:0}: Error finding container 0d3dcddd3b2e4e39dba90039c8f44372cff5f483a369122ce0aa0adfb37daf6a: Status 404 returned error can't find the container with id 0d3dcddd3b2e4e39dba90039c8f44372cff5f483a369122ce0aa0adfb37daf6a
Oct 30 18:42:36.402216 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:36.402079    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:36.402455 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:36.402354    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:36.402455 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:36.402416    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:37.014544 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.014498    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2" event={"ID":"2b704d30-26f4-4f7f-9b78-aece8de84de0","Type":"ContainerStarted","Data":"9905403da43802713fc5bae049dd0ab6aa3a5754ca416ff3be96ef421ace27f2"}
Oct 30 18:42:37.015053 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.014561    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2" event={"ID":"2b704d30-26f4-4f7f-9b78-aece8de84de0","Type":"ContainerStarted","Data":"0d3dcddd3b2e4e39dba90039c8f44372cff5f483a369122ce0aa0adfb37daf6a"}
Oct 30 18:42:37.044842 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.044745    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2" podStartSLOduration=2.044723615 podStartE2EDuration="2.044723615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:37.043349332 +0000 UTC m=+570.911767662" watchObservedRunningTime="2026-10-30 18:42:37.044723615 +0000 UTC m=+570.913141944"
Oct 30 18:42:37.072923 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.072850    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:37.079825 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.079774    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:37.261476 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.261431    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"]
Oct 30 18:42:37.265395 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.265305    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.272916 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.272882    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9aded813-openshift-default-dockercfg-4d4hw\""
Oct 30 18:42:37.283577 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.283516    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"]
Oct 30 18:42:37.368780 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.368725    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-workload-certs\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.369161 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.369075    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-envoy\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.369414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.369201    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-token\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.369414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.369324    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-podinfo\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.369414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.369375    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-data\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.369609 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.369433    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-workload-socket\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.369609 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.369474    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vbgc\" (UniqueName: \"kubernetes.io/projected/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-kube-api-access-7vbgc\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.369609 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.369506    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istiod-ca-cert\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.369609 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.369596    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-credential-socket\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.471232 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.471171    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-token\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.471483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.471270    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-podinfo\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.471483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.471312    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-data\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.471483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.471347    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-workload-socket\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.471483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.471387    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vbgc\" (UniqueName: \"kubernetes.io/projected/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-kube-api-access-7vbgc\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.471483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.471414    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istiod-ca-cert\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.471483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.471474    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-credential-socket\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.471766 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.471565    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-workload-certs\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.471766 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.471600    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-envoy\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.471965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.471937    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-data\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.472116 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.472088    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-credential-socket\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.472240 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.472032    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-workload-certs\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.472240 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.472178    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-workload-socket\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.472655 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.472631    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istiod-ca-cert\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.474524 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.474494    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-envoy\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.474752 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.474732    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-podinfo\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.485642 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.485611    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-token\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.485830 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.485765    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vbgc\" (UniqueName: \"kubernetes.io/projected/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-kube-api-access-7vbgc\") pod \"e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") " pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.583259 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.583215    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:37.896957 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:42:37.896912    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2f3a26_d0d7_41af_9a12_b020d1a9e82d.slice/crio-49c8e6078aff098103e1d053b114690d2c0edefd6df6ee8d04ea723d141f5bbc WatchSource:0}: Error finding container 49c8e6078aff098103e1d053b114690d2c0edefd6df6ee8d04ea723d141f5bbc: Status 404 returned error can't find the container with id 49c8e6078aff098103e1d053b114690d2c0edefd6df6ee8d04ea723d141f5bbc
Oct 30 18:42:37.899773 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.899724    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:37.899911 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.899805    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:37.899911 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.899839    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:37.904726 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:37.904688    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"]
Oct 30 18:42:38.024396 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:38.024336    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp" event={"ID":"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d","Type":"ContainerStarted","Data":"0d2ed977f1b12ea4e522e4ab5d9521c2b10614cfea4eb43cf3f9da6c29215413"}
Oct 30 18:42:38.025017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:38.024418    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp" event={"ID":"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d","Type":"ContainerStarted","Data":"49c8e6078aff098103e1d053b114690d2c0edefd6df6ee8d04ea723d141f5bbc"}
Oct 30 18:42:38.025017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:38.024677    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:38.026021 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:38.025997    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:42:38.066285 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:38.066197    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp" podStartSLOduration=1.066165146 podStartE2EDuration="1.066165146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:38.064363983 +0000 UTC m=+571.932782329" watchObservedRunningTime="2026-10-30 18:42:38.066165146 +0000 UTC m=+571.934583468"
Oct 30 18:42:38.583788 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:38.583737    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:38.590029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:38.589993    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:39.030311 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:39.030208    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:39.031460 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:39.031431    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:42:39.947632 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:39.947588    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"]
Oct 30 18:42:39.951438 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:39.951411    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"
Oct 30 18:42:39.963892 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:39.963856    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d7289525\""
Oct 30 18:42:39.966153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:39.966088    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"]
Oct 30 18:42:39.999406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:39.999359    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a92661a-02c0-4788-8f95-0fbd601cabe8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7\" (UID: \"8a92661a-02c0-4788-8f95-0fbd601cabe8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"
Oct 30 18:42:39.999616 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:39.999549    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-528mv\" (UniqueName: \"kubernetes.io/projected/8a92661a-02c0-4788-8f95-0fbd601cabe8-kube-api-access-528mv\") pod \"payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7\" (UID: \"8a92661a-02c0-4788-8f95-0fbd601cabe8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"
Oct 30 18:42:40.100912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.100862    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528mv\" (UniqueName: \"kubernetes.io/projected/8a92661a-02c0-4788-8f95-0fbd601cabe8-kube-api-access-528mv\") pod \"payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7\" (UID: \"8a92661a-02c0-4788-8f95-0fbd601cabe8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"
Oct 30 18:42:40.101750 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.101710    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a92661a-02c0-4788-8f95-0fbd601cabe8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7\" (UID: \"8a92661a-02c0-4788-8f95-0fbd601cabe8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"
Oct 30 18:42:40.106432 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.105106    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a92661a-02c0-4788-8f95-0fbd601cabe8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7\" (UID: \"8a92661a-02c0-4788-8f95-0fbd601cabe8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"
Oct 30 18:42:40.132170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.129330    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-528mv\" (UniqueName: \"kubernetes.io/projected/8a92661a-02c0-4788-8f95-0fbd601cabe8-kube-api-access-528mv\") pod \"payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7\" (UID: \"8a92661a-02c0-4788-8f95-0fbd601cabe8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"
Oct 30 18:42:40.267331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.267222    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"
Oct 30 18:42:40.450791 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.450751    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"]
Oct 30 18:42:40.456370 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.456332    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"
Oct 30 18:42:40.461434 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.461398    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"]
Oct 30 18:42:40.465030 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.464981    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"
Oct 30 18:42:40.468748 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.468713    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-78934afe\""
Oct 30 18:42:40.468937 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.468882    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-78934afe-dockercfg-cf4rx\""
Oct 30 18:42:40.482960 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.482898    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"]
Oct 30 18:42:40.485071 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.485036    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"]
Oct 30 18:42:40.506188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.506142    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26724afc-c0be-4771-a468-1ab2d4d08e52-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p\" (UID: \"26724afc-c0be-4771-a468-1ab2d4d08e52\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"
Oct 30 18:42:40.506406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.506339    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtptl\" (UniqueName: \"kubernetes.io/projected/26724afc-c0be-4771-a468-1ab2d4d08e52-kube-api-access-jtptl\") pod \"payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p\" (UID: \"26724afc-c0be-4771-a468-1ab2d4d08e52\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"
Oct 30 18:42:40.506406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.506394    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424cc5a4-a1f7-4d36-ae31-497606a79c16-plugins-config-volume\") pod \"payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd\" (UID: \"424cc5a4-a1f7-4d36-ae31-497606a79c16\") " pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"
Oct 30 18:42:40.506564 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.506542    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zz6x\" (UniqueName: \"kubernetes.io/projected/424cc5a4-a1f7-4d36-ae31-497606a79c16-kube-api-access-6zz6x\") pod \"payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd\" (UID: \"424cc5a4-a1f7-4d36-ae31-497606a79c16\") " pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"
Oct 30 18:42:40.570093 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.569977    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"]
Oct 30 18:42:40.574224 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:42:40.574169    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a92661a_02c0_4788_8f95_0fbd601cabe8.slice/crio-78c9dd1e30521b05d90fc0e621c39c213d433dc0209137e2c575f2155fb79578 WatchSource:0}: Error finding container 78c9dd1e30521b05d90fc0e621c39c213d433dc0209137e2c575f2155fb79578: Status 404 returned error can't find the container with id 78c9dd1e30521b05d90fc0e621c39c213d433dc0209137e2c575f2155fb79578
Oct 30 18:42:40.622160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.610214    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zz6x\" (UniqueName: \"kubernetes.io/projected/424cc5a4-a1f7-4d36-ae31-497606a79c16-kube-api-access-6zz6x\") pod \"payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd\" (UID: \"424cc5a4-a1f7-4d36-ae31-497606a79c16\") " pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"
Oct 30 18:42:40.622160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.610278    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26724afc-c0be-4771-a468-1ab2d4d08e52-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p\" (UID: \"26724afc-c0be-4771-a468-1ab2d4d08e52\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"
Oct 30 18:42:40.622160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.610407    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtptl\" (UniqueName: \"kubernetes.io/projected/26724afc-c0be-4771-a468-1ab2d4d08e52-kube-api-access-jtptl\") pod \"payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p\" (UID: \"26724afc-c0be-4771-a468-1ab2d4d08e52\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"
Oct 30 18:42:40.622160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.610467    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424cc5a4-a1f7-4d36-ae31-497606a79c16-plugins-config-volume\") pod \"payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd\" (UID: \"424cc5a4-a1f7-4d36-ae31-497606a79c16\") " pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"
Oct 30 18:42:40.622160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.611626    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424cc5a4-a1f7-4d36-ae31-497606a79c16-plugins-config-volume\") pod \"payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd\" (UID: \"424cc5a4-a1f7-4d36-ae31-497606a79c16\") " pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"
Oct 30 18:42:40.622160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.612153    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26724afc-c0be-4771-a468-1ab2d4d08e52-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p\" (UID: \"26724afc-c0be-4771-a468-1ab2d4d08e52\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"
Oct 30 18:42:40.650563 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.646155    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtptl\" (UniqueName: \"kubernetes.io/projected/26724afc-c0be-4771-a468-1ab2d4d08e52-kube-api-access-jtptl\") pod \"payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p\" (UID: \"26724afc-c0be-4771-a468-1ab2d4d08e52\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"
Oct 30 18:42:40.671197 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.668105    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zz6x\" (UniqueName: \"kubernetes.io/projected/424cc5a4-a1f7-4d36-ae31-497606a79c16-kube-api-access-6zz6x\") pod \"payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd\" (UID: \"424cc5a4-a1f7-4d36-ae31-497606a79c16\") " pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"
Oct 30 18:42:40.775015 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.774958    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"
Oct 30 18:42:40.784729 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:40.784687    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"
Oct 30 18:42:41.043812 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.043759    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7" event={"ID":"8a92661a-02c0-4788-8f95-0fbd601cabe8","Type":"ContainerStarted","Data":"fa805a16fa85d8cf04b8cbe7aa36c5d899da3ffa2baa28cc4d49e5aaef7e5a89"}
Oct 30 18:42:41.044073 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.043875    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7" event={"ID":"8a92661a-02c0-4788-8f95-0fbd601cabe8","Type":"ContainerStarted","Data":"78c9dd1e30521b05d90fc0e621c39c213d433dc0209137e2c575f2155fb79578"}
Oct 30 18:42:41.045817 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.045785    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"
Oct 30 18:42:41.071424 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.071246    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"]
Oct 30 18:42:41.072810 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.072746    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7" podStartSLOduration=2.072723752 podStartE2EDuration="2.072723752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:41.07230507 +0000 UTC m=+574.940723426" watchObservedRunningTime="2026-10-30 18:42:41.072723752 +0000 UTC m=+574.941142081"
Oct 30 18:42:41.074404 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:42:41.074371    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424cc5a4_a1f7_4d36_ae31_497606a79c16.slice/crio-bc8bdea1a83b0514785966873ddf945833662012ee6f9b081c68a96a50ccee00 WatchSource:0}: Error finding container bc8bdea1a83b0514785966873ddf945833662012ee6f9b081c68a96a50ccee00: Status 404 returned error can't find the container with id bc8bdea1a83b0514785966873ddf945833662012ee6f9b081c68a96a50ccee00
Oct 30 18:42:41.094931 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.094893    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"]
Oct 30 18:42:41.097482 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:42:41.097447    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26724afc_c0be_4771_a468_1ab2d4d08e52.slice/crio-fd6526f7b85f91130cf2d0ac736ec2928c26d9d3ad9f9f7b747ab37f1b70bdf7 WatchSource:0}: Error finding container fd6526f7b85f91130cf2d0ac736ec2928c26d9d3ad9f9f7b747ab37f1b70bdf7: Status 404 returned error can't find the container with id fd6526f7b85f91130cf2d0ac736ec2928c26d9d3ad9f9f7b747ab37f1b70bdf7
Oct 30 18:42:41.630882 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.630834    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"]
Oct 30 18:42:41.634885 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.634852    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:41.644384 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.644352    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-78934afe-serving-cert\""
Oct 30 18:42:41.647533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.647498    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"]
Oct 30 18:42:41.725110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.725067    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ce611c-7b51-4ad0-9577-c9d1c1630836-openshift-service-ca\") pod \"maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:41.725367 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.725217    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15ce611c-7b51-4ad0-9577-c9d1c1630836-metrics-certs\") pod \"maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:41.725367 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.725250    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76jf9\" (UniqueName: \"kubernetes.io/projected/15ce611c-7b51-4ad0-9577-c9d1c1630836-kube-api-access-76jf9\") pod \"maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:41.725367 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.725316    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ce611c-7b51-4ad0-9577-c9d1c1630836-maas-api-tls\") pod \"maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:41.826610 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.826544    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15ce611c-7b51-4ad0-9577-c9d1c1630836-metrics-certs\") pod \"maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:41.826610 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.826608    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76jf9\" (UniqueName: \"kubernetes.io/projected/15ce611c-7b51-4ad0-9577-c9d1c1630836-kube-api-access-76jf9\") pod \"maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:41.826975 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.826667    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ce611c-7b51-4ad0-9577-c9d1c1630836-maas-api-tls\") pod \"maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:41.826975 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.826823    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ce611c-7b51-4ad0-9577-c9d1c1630836-openshift-service-ca\") pod \"maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:41.827642 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.827548    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ce611c-7b51-4ad0-9577-c9d1c1630836-openshift-service-ca\") pod \"maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:41.829826 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.829793    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ce611c-7b51-4ad0-9577-c9d1c1630836-maas-api-tls\") pod \"maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:41.829938 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.829793    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15ce611c-7b51-4ad0-9577-c9d1c1630836-metrics-certs\") pod \"maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:41.841143 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.841094    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76jf9\" (UniqueName: \"kubernetes.io/projected/15ce611c-7b51-4ad0-9577-c9d1c1630836-kube-api-access-76jf9\") pod \"maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:41.951637 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:41.951516    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:42.052735 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:42.052684    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd" event={"ID":"424cc5a4-a1f7-4d36-ae31-497606a79c16","Type":"ContainerStarted","Data":"9f778f10ff9ab0ed2d667b47e86a900909a19348ef73f76d657dd427812c13b8"}
Oct 30 18:42:42.053022 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:42.052763    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd" event={"ID":"424cc5a4-a1f7-4d36-ae31-497606a79c16","Type":"ContainerStarted","Data":"bc8bdea1a83b0514785966873ddf945833662012ee6f9b081c68a96a50ccee00"}
Oct 30 18:42:42.053849 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:42.053801    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"
Oct 30 18:42:42.056052 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:42.056017    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p" event={"ID":"26724afc-c0be-4771-a468-1ab2d4d08e52","Type":"ContainerStarted","Data":"633b154f867d42b92341c3ed2111408412983157855e50788cee2605cdf8313a"}
Oct 30 18:42:42.056280 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:42.056080    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p" event={"ID":"26724afc-c0be-4771-a468-1ab2d4d08e52","Type":"ContainerStarted","Data":"fd6526f7b85f91130cf2d0ac736ec2928c26d9d3ad9f9f7b747ab37f1b70bdf7"}
Oct 30 18:42:42.056280 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:42.056270    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"
Oct 30 18:42:42.080702 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:42.080606    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd" podStartSLOduration=2.080580479 podStartE2EDuration="2.080580479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:42.07917239 +0000 UTC m=+575.947590735" watchObservedRunningTime="2026-10-30 18:42:42.080580479 +0000 UTC m=+575.948998809"
Oct 30 18:42:42.104849 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:42.104753    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p" podStartSLOduration=2.104728245 podStartE2EDuration="2.104728245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:42.102924482 +0000 UTC m=+575.971342814" watchObservedRunningTime="2026-10-30 18:42:42.104728245 +0000 UTC m=+575.973146646"
Oct 30 18:42:42.280688 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:42.279880    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"]
Oct 30 18:42:43.064909 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:43.064766    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm" event={"ID":"15ce611c-7b51-4ad0-9577-c9d1c1630836","Type":"ContainerStarted","Data":"ff79e778e6d0635e7c4918b2f9f3cb5798bde151a9ef09c80f0b952ae1257e4a"}
Oct 30 18:42:43.064909 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:43.064854    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm" event={"ID":"15ce611c-7b51-4ad0-9577-c9d1c1630836","Type":"ContainerStarted","Data":"3b6103cd314a5b70bea98c65ece88841d1965f320242756ccb198c1d77c29f7d"}
Oct 30 18:42:43.088865 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:43.088766    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm" podStartSLOduration=1.6176365270000002 podStartE2EDuration="2.088745925s" totalImagesPullingTime="471.109398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:43.086548005 +0000 UTC m=+576.954966335" watchObservedRunningTime="2026-10-30 18:42:43.088745925 +0000 UTC m=+576.957164255"
Oct 30 18:42:43.372614 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:43.372546    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:42:44.076027 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:44.075988    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:44.946046 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:44.945998    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"]
Oct 30 18:42:44.950667 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:44.950625    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:44.967803 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:44.967746    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-metrics-certs\") pod \"maas-api-e2e-ait-9aded813-598bfc4c-k2t5v\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:44.968030 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:44.967820    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/360aad55-05a3-4b11-acb5-c15c91f83b49-openshift-service-ca\") pod \"maas-api-e2e-ait-9aded813-598bfc4c-k2t5v\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:44.968089 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:44.968055    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9t5h\" (UniqueName: \"kubernetes.io/projected/360aad55-05a3-4b11-acb5-c15c91f83b49-kube-api-access-h9t5h\") pod \"maas-api-e2e-ait-9aded813-598bfc4c-k2t5v\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:44.968210 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:44.968193    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-maas-api-tls\") pod \"maas-api-e2e-ait-9aded813-598bfc4c-k2t5v\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:44.977651 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:44.977573    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"]
Oct 30 18:42:44.987214 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:44.987181    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9aded813-serving-cert\""
Oct 30 18:42:45.069261 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:45.069188    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-metrics-certs\") pod \"maas-api-e2e-ait-9aded813-598bfc4c-k2t5v\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:45.069571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:45.069306    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/360aad55-05a3-4b11-acb5-c15c91f83b49-openshift-service-ca\") pod \"maas-api-e2e-ait-9aded813-598bfc4c-k2t5v\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:45.069571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:45.069401    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9t5h\" (UniqueName: \"kubernetes.io/projected/360aad55-05a3-4b11-acb5-c15c91f83b49-kube-api-access-h9t5h\") pod \"maas-api-e2e-ait-9aded813-598bfc4c-k2t5v\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:45.069571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:45.069460    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-maas-api-tls\") pod \"maas-api-e2e-ait-9aded813-598bfc4c-k2t5v\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:45.069716 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:42:45.069624    4372 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-serving-cert: secret "maas-api-e2e-ait-9aded813-serving-cert" not found
Oct 30 18:42:45.069766 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:42:45.069719    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-maas-api-tls podName:360aad55-05a3-4b11-acb5-c15c91f83b49 nodeName:}" failed. No retries permitted until 2026-10-30 18:42:45.569691016 +0000 UTC m=+579.438109327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-maas-api-tls") pod "maas-api-e2e-ait-9aded813-598bfc4c-k2t5v" (UID: "360aad55-05a3-4b11-acb5-c15c91f83b49") : secret "maas-api-e2e-ait-9aded813-serving-cert" not found
Oct 30 18:42:45.070344 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:45.070269    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/360aad55-05a3-4b11-acb5-c15c91f83b49-openshift-service-ca\") pod \"maas-api-e2e-ait-9aded813-598bfc4c-k2t5v\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:45.080184 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:45.073404    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-metrics-certs\") pod \"maas-api-e2e-ait-9aded813-598bfc4c-k2t5v\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:45.091186 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:45.090426    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9t5h\" (UniqueName: \"kubernetes.io/projected/360aad55-05a3-4b11-acb5-c15c91f83b49-kube-api-access-h9t5h\") pod \"maas-api-e2e-ait-9aded813-598bfc4c-k2t5v\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:45.574815 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:45.574761    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-maas-api-tls\") pod \"maas-api-e2e-ait-9aded813-598bfc4c-k2t5v\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:45.577905 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:45.577877    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-maas-api-tls\") pod \"maas-api-e2e-ait-9aded813-598bfc4c-k2t5v\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:45.865617 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:45.865495    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:46.156985 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:46.156919    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"]
Oct 30 18:42:46.159895 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:42:46.159854    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360aad55_05a3_4b11_acb5_c15c91f83b49.slice/crio-63d5410be1eb86fdc12093059948fb34605342e7a3000b09572a971908ab43da WatchSource:0}: Error finding container 63d5410be1eb86fdc12093059948fb34605342e7a3000b09572a971908ab43da: Status 404 returned error can't find the container with id 63d5410be1eb86fdc12093059948fb34605342e7a3000b09572a971908ab43da
Oct 30 18:42:47.118709 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:47.118603    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v" event={"ID":"360aad55-05a3-4b11-acb5-c15c91f83b49","Type":"ContainerStarted","Data":"f9de4a992f3de5e7ae4062ebcf8bbcd24391a135b0673b1511c2d1967e9dad35"}
Oct 30 18:42:47.118709 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:47.118676    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:47.118709 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:47.118693    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v" event={"ID":"360aad55-05a3-4b11-acb5-c15c91f83b49","Type":"ContainerStarted","Data":"63d5410be1eb86fdc12093059948fb34605342e7a3000b09572a971908ab43da"}
Oct 30 18:42:47.162898 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:47.162819    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v" podStartSLOduration=2.757222493 podStartE2EDuration="3.162791188s" totalImagesPullingTime="405.568695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:47.16019289 +0000 UTC m=+581.028611214" watchObservedRunningTime="2026-10-30 18:42:47.162791188 +0000 UTC m=+581.031209812"
Oct 30 18:42:47.403461 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:47.403338    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:42:50.128467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:50.128419    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:42:53.075115 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:53.075063    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"
Oct 30 18:42:53.101628 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:53.101588    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"
Oct 30 18:42:53.151208 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:53.151163    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:42:54.115477 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:54.115421    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"
Oct 30 18:42:55.382377 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:55.382301    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:42:57.705519 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:42:57.705464    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:42:59.257786 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:59.257656    4372 generic.go:356] "Generic (PLEG): container finished" podID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" containerID="0d2ed977f1b12ea4e522e4ab5d9521c2b10614cfea4eb43cf3f9da6c29215413" exitCode=0
Oct 30 18:42:59.257786 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:59.257739    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp" event={"ID":"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d","Type":"ContainerDied","Data":"0d2ed977f1b12ea4e522e4ab5d9521c2b10614cfea4eb43cf3f9da6c29215413"}
Oct 30 18:42:59.258458 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:59.258249    4372 scope.go:122] "RemoveContainer" containerID="0d2ed977f1b12ea4e522e4ab5d9521c2b10614cfea4eb43cf3f9da6c29215413"
Oct 30 18:42:59.259766 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:59.259731    4372 generic.go:356] "Generic (PLEG): container finished" podID="2b704d30-26f4-4f7f-9b78-aece8de84de0" containerID="9905403da43802713fc5bae049dd0ab6aa3a5754ca416ff3be96ef421ace27f2" exitCode=0
Oct 30 18:42:59.259917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:59.259811    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2" event={"ID":"2b704d30-26f4-4f7f-9b78-aece8de84de0","Type":"ContainerDied","Data":"9905403da43802713fc5bae049dd0ab6aa3a5754ca416ff3be96ef421ace27f2"}
Oct 30 18:42:59.260040 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:59.260002    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:59.260187 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:59.260162    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:59.260249 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:59.260230    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:59.260355 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:59.260341    4372 scope.go:122] "RemoveContainer" containerID="9905403da43802713fc5bae049dd0ab6aa3a5754ca416ff3be96ef421ace27f2"
Oct 30 18:42:59.261682 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:59.261651    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:59.261787 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:59.261725    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:42:59.261787 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:42:59.261765    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:00.271579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:00.269313    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2" event={"ID":"2b704d30-26f4-4f7f-9b78-aece8de84de0","Type":"ContainerStarted","Data":"8aa4c1004dd7408374ee147e3c9bc8c068e8ac5482a28f761f251000aabc9d31"}
Oct 30 18:43:00.272216 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:00.271772    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp" event={"ID":"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d","Type":"ContainerStarted","Data":"a87d81511bfa54c369c0f7d0a984493bed2e8b54702e5288c9979deb2b8fb084"}
Oct 30 18:43:00.590165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:00.586817    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:43:00.590165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:00.587006    4372 patch_prober.go:29] interesting pod/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 18:43:00.590165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:00.587062    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp" podUID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 18:43:01.072706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:01.072513    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:43:01.073307 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:01.073272    4372 patch_prober.go:29] interesting pod/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 18:43:01.073550 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:01.073350    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2" podUID="2b704d30-26f4-4f7f-9b78-aece8de84de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 18:43:01.593157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:01.591745    4372 patch_prober.go:29] interesting pod/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 18:43:01.593157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:01.591835    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp" podUID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 18:43:02.073000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:02.072897    4372 patch_prober.go:29] interesting pod/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 18:43:02.073000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:02.072963    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2" podUID="2b704d30-26f4-4f7f-9b78-aece8de84de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 18:43:02.602163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:02.598860    4372 patch_prober.go:29] interesting pod/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 18:43:02.602163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:02.598955    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp" podUID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 18:43:03.073048 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:03.072920    4372 patch_prober.go:29] interesting pod/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 18:43:03.073048 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:03.072986    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2" podUID="2b704d30-26f4-4f7f-9b78-aece8de84de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 18:43:03.586884 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:03.586830    4372 patch_prober.go:29] interesting pod/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 18:43:03.587384 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:03.587338    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp" podUID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 18:43:04.072839 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:04.072736    4372 patch_prober.go:29] interesting pod/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 18:43:04.072839 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:04.072809    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2" podUID="2b704d30-26f4-4f7f-9b78-aece8de84de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 18:43:04.586164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:04.584869    4372 patch_prober.go:29] interesting pod/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 18:43:04.586164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:04.584971    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp" podUID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 18:43:05.072680 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:05.072570    4372 patch_prober.go:29] interesting pod/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 18:43:05.072680 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:05.072645    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2" podUID="2b704d30-26f4-4f7f-9b78-aece8de84de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 18:43:05.588000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:05.587950    4372 patch_prober.go:29] interesting pod/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 18:43:05.613420 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:05.612803    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp" podUID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 18:43:06.086052 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:06.086011    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:43:06.086052 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:06.088283    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:43:06.094559 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:06.094527    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:43:06.590076 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:06.590020    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:43:06.590644 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:06.590614    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:43:06.591869 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:06.591845    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:43:09.344835 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:09.344789    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"]
Oct 30 18:43:09.543663 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:09.543605    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"]
Oct 30 18:43:09.543969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:09.543937    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm" podUID="15ce611c-7b51-4ad0-9577-c9d1c1630836" containerName="maas-api" containerID="cri-o://ff79e778e6d0635e7c4918b2f9f3cb5798bde151a9ef09c80f0b952ae1257e4a" gracePeriod=30
Oct 30 18:43:09.621447 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:09.621320    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"]
Oct 30 18:43:09.644022 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:09.643972    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"]
Oct 30 18:43:09.882747 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:09.882609    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:43:10.031642 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.031588    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15ce611c-7b51-4ad0-9577-c9d1c1630836-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ce611c-7b51-4ad0-9577-c9d1c1630836-openshift-service-ca\") pod \"15ce611c-7b51-4ad0-9577-c9d1c1630836\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") "
Oct 30 18:43:10.032054 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.031726    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15ce611c-7b51-4ad0-9577-c9d1c1630836-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15ce611c-7b51-4ad0-9577-c9d1c1630836-metrics-certs\") pod \"15ce611c-7b51-4ad0-9577-c9d1c1630836\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") "
Oct 30 18:43:10.032054 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.031786    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ce611c-7b51-4ad0-9577-c9d1c1630836-kube-api-access-76jf9\" (UniqueName: \"kubernetes.io/projected/15ce611c-7b51-4ad0-9577-c9d1c1630836-kube-api-access-76jf9\") pod \"15ce611c-7b51-4ad0-9577-c9d1c1630836\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") "
Oct 30 18:43:10.032054 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.031987    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15ce611c-7b51-4ad0-9577-c9d1c1630836-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ce611c-7b51-4ad0-9577-c9d1c1630836-maas-api-tls\") pod \"15ce611c-7b51-4ad0-9577-c9d1c1630836\" (UID: \"15ce611c-7b51-4ad0-9577-c9d1c1630836\") "
Oct 30 18:43:10.032266 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.032190    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ce611c-7b51-4ad0-9577-c9d1c1630836-openshift-service-ca" pod "15ce611c-7b51-4ad0-9577-c9d1c1630836" (UID: "15ce611c-7b51-4ad0-9577-c9d1c1630836"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:10.032495 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.032465    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ce611c-7b51-4ad0-9577-c9d1c1630836-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:10.035161 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.035068    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15ce611c-7b51-4ad0-9577-c9d1c1630836-maas-api-tls" pod "15ce611c-7b51-4ad0-9577-c9d1c1630836" (UID: "15ce611c-7b51-4ad0-9577-c9d1c1630836"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:43:10.035334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.035189    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15ce611c-7b51-4ad0-9577-c9d1c1630836-metrics-certs" pod "15ce611c-7b51-4ad0-9577-c9d1c1630836" (UID: "15ce611c-7b51-4ad0-9577-c9d1c1630836"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:43:10.035334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.035185    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ce611c-7b51-4ad0-9577-c9d1c1630836-kube-api-access-76jf9" pod "15ce611c-7b51-4ad0-9577-c9d1c1630836" (UID: "15ce611c-7b51-4ad0-9577-c9d1c1630836"). InnerVolumeSpecName "kube-api-access-76jf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:10.133678 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.133551    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15ce611c-7b51-4ad0-9577-c9d1c1630836-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:10.133678 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.133592    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15ce611c-7b51-4ad0-9577-c9d1c1630836-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:10.133678 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.133606    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76jf9\" (UniqueName: \"kubernetes.io/projected/15ce611c-7b51-4ad0-9577-c9d1c1630836-kube-api-access-76jf9\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:10.221052 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.221002    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"]
Oct 30 18:43:10.221448 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.221412    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2" podUID="2b704d30-26f4-4f7f-9b78-aece8de84de0" containerName="istio-proxy" containerID="cri-o://8aa4c1004dd7408374ee147e3c9bc8c068e8ac5482a28f761f251000aabc9d31" gracePeriod=30
Oct 30 18:43:10.398020 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.397874    4372 generic.go:356] "Generic (PLEG): container finished" podID="15ce611c-7b51-4ad0-9577-c9d1c1630836" containerID="ff79e778e6d0635e7c4918b2f9f3cb5798bde151a9ef09c80f0b952ae1257e4a" exitCode=0
Oct 30 18:43:10.398020 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.397936    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"
Oct 30 18:43:10.398020 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.397963    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm" event={"ID":"15ce611c-7b51-4ad0-9577-c9d1c1630836","Type":"ContainerDied","Data":"ff79e778e6d0635e7c4918b2f9f3cb5798bde151a9ef09c80f0b952ae1257e4a"}
Oct 30 18:43:10.398745 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.398035    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm" event={"ID":"15ce611c-7b51-4ad0-9577-c9d1c1630836","Type":"ContainerDied","Data":"3b6103cd314a5b70bea98c65ece88841d1965f320242756ccb198c1d77c29f7d"}
Oct 30 18:43:10.398745 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.398069    4372 scope.go:122] "RemoveContainer" containerID="ff79e778e6d0635e7c4918b2f9f3cb5798bde151a9ef09c80f0b952ae1257e4a"
Oct 30 18:43:10.426986 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.416636    4372 scope.go:122] "RemoveContainer" containerID="ff79e778e6d0635e7c4918b2f9f3cb5798bde151a9ef09c80f0b952ae1257e4a"
Oct 30 18:43:10.426986 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:43:10.418764    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff79e778e6d0635e7c4918b2f9f3cb5798bde151a9ef09c80f0b952ae1257e4a\": container with ID starting with ff79e778e6d0635e7c4918b2f9f3cb5798bde151a9ef09c80f0b952ae1257e4a not found: ID does not exist" containerID="ff79e778e6d0635e7c4918b2f9f3cb5798bde151a9ef09c80f0b952ae1257e4a"
Oct 30 18:43:10.426986 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.418831    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff79e778e6d0635e7c4918b2f9f3cb5798bde151a9ef09c80f0b952ae1257e4a"} err="failed to get container status \"ff79e778e6d0635e7c4918b2f9f3cb5798bde151a9ef09c80f0b952ae1257e4a\": rpc error: code = NotFound desc = could not find container \"ff79e778e6d0635e7c4918b2f9f3cb5798bde151a9ef09c80f0b952ae1257e4a\": container with ID starting with ff79e778e6d0635e7c4918b2f9f3cb5798bde151a9ef09c80f0b952ae1257e4a not found: ID does not exist"
Oct 30 18:43:10.439042 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.438995    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"]
Oct 30 18:43:10.446931 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.446881    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-78934afe-84bdc845fd-sn7dm"]
Oct 30 18:43:10.982035 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.981989    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"]
Oct 30 18:43:10.982344 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:10.982308    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v" podUID="360aad55-05a3-4b11-acb5-c15c91f83b49" containerName="maas-api" containerID="cri-o://f9de4a992f3de5e7ae4062ebcf8bbcd24391a135b0673b1511c2d1967e9dad35" gracePeriod=30
Oct 30 18:43:11.289812 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.289780    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:43:11.406011 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.405963    4372 generic.go:356] "Generic (PLEG): container finished" podID="360aad55-05a3-4b11-acb5-c15c91f83b49" containerID="f9de4a992f3de5e7ae4062ebcf8bbcd24391a135b0673b1511c2d1967e9dad35" exitCode=0
Oct 30 18:43:11.406620 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.406079    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"
Oct 30 18:43:11.406620 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.406104    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v" event={"ID":"360aad55-05a3-4b11-acb5-c15c91f83b49","Type":"ContainerDied","Data":"f9de4a992f3de5e7ae4062ebcf8bbcd24391a135b0673b1511c2d1967e9dad35"}
Oct 30 18:43:11.406620 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.406199    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v" event={"ID":"360aad55-05a3-4b11-acb5-c15c91f83b49","Type":"ContainerDied","Data":"63d5410be1eb86fdc12093059948fb34605342e7a3000b09572a971908ab43da"}
Oct 30 18:43:11.406620 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.406231    4372 scope.go:122] "RemoveContainer" containerID="f9de4a992f3de5e7ae4062ebcf8bbcd24391a135b0673b1511c2d1967e9dad35"
Oct 30 18:43:11.417364 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.417336    4372 scope.go:122] "RemoveContainer" containerID="f9de4a992f3de5e7ae4062ebcf8bbcd24391a135b0673b1511c2d1967e9dad35"
Oct 30 18:43:11.417780 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:43:11.417752    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9de4a992f3de5e7ae4062ebcf8bbcd24391a135b0673b1511c2d1967e9dad35\": container with ID starting with f9de4a992f3de5e7ae4062ebcf8bbcd24391a135b0673b1511c2d1967e9dad35 not found: ID does not exist" containerID="f9de4a992f3de5e7ae4062ebcf8bbcd24391a135b0673b1511c2d1967e9dad35"
Oct 30 18:43:11.417841 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.417798    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9de4a992f3de5e7ae4062ebcf8bbcd24391a135b0673b1511c2d1967e9dad35"} err="failed to get container status \"f9de4a992f3de5e7ae4062ebcf8bbcd24391a135b0673b1511c2d1967e9dad35\": rpc error: code = NotFound desc = could not find container \"f9de4a992f3de5e7ae4062ebcf8bbcd24391a135b0673b1511c2d1967e9dad35\": container with ID starting with f9de4a992f3de5e7ae4062ebcf8bbcd24391a135b0673b1511c2d1967e9dad35 not found: ID does not exist"
Oct 30 18:43:11.464328 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.464274    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360aad55-05a3-4b11-acb5-c15c91f83b49-kube-api-access-h9t5h\" (UniqueName: \"kubernetes.io/projected/360aad55-05a3-4b11-acb5-c15c91f83b49-kube-api-access-h9t5h\") pod \"360aad55-05a3-4b11-acb5-c15c91f83b49\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") "
Oct 30 18:43:11.464610 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.464400    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-maas-api-tls\") pod \"360aad55-05a3-4b11-acb5-c15c91f83b49\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") "
Oct 30 18:43:11.464610 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.464452    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/360aad55-05a3-4b11-acb5-c15c91f83b49-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/360aad55-05a3-4b11-acb5-c15c91f83b49-openshift-service-ca\") pod \"360aad55-05a3-4b11-acb5-c15c91f83b49\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") "
Oct 30 18:43:11.464610 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.464492    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-metrics-certs\" (UniqueName: \"kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-metrics-certs\") pod \"360aad55-05a3-4b11-acb5-c15c91f83b49\" (UID: \"360aad55-05a3-4b11-acb5-c15c91f83b49\") "
Oct 30 18:43:11.464920 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.464886    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/360aad55-05a3-4b11-acb5-c15c91f83b49-openshift-service-ca" pod "360aad55-05a3-4b11-acb5-c15c91f83b49" (UID: "360aad55-05a3-4b11-acb5-c15c91f83b49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:11.467280 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.467235    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-maas-api-tls" pod "360aad55-05a3-4b11-acb5-c15c91f83b49" (UID: "360aad55-05a3-4b11-acb5-c15c91f83b49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:43:11.467447 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.467294    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-metrics-certs" pod "360aad55-05a3-4b11-acb5-c15c91f83b49" (UID: "360aad55-05a3-4b11-acb5-c15c91f83b49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:43:11.467447 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.467302    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360aad55-05a3-4b11-acb5-c15c91f83b49-kube-api-access-h9t5h" pod "360aad55-05a3-4b11-acb5-c15c91f83b49" (UID: "360aad55-05a3-4b11-acb5-c15c91f83b49"). InnerVolumeSpecName "kube-api-access-h9t5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:11.565838 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.565703    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:11.565838 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.565765    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/360aad55-05a3-4b11-acb5-c15c91f83b49-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:11.565838 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.565781    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/360aad55-05a3-4b11-acb5-c15c91f83b49-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:11.565838 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.565794    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9t5h\" (UniqueName: \"kubernetes.io/projected/360aad55-05a3-4b11-acb5-c15c91f83b49-kube-api-access-h9t5h\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:11.632722 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.632672    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"]
Oct 30 18:43:11.633401 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.633377    4372 cpu_manager.go:414] "containerMap: removing container" podUID="360aad55-05a3-4b11-acb5-c15c91f83b49" containerName="maas-api"
Oct 30 18:43:11.633448 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.633406    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="360aad55-05a3-4b11-acb5-c15c91f83b49" containerName="maas-api"
Oct 30 18:43:11.633485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.633447    4372 cpu_manager.go:414] "containerMap: removing container" podUID="15ce611c-7b51-4ad0-9577-c9d1c1630836" containerName="maas-api"
Oct 30 18:43:11.633485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.633455    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ce611c-7b51-4ad0-9577-c9d1c1630836" containerName="maas-api"
Oct 30 18:43:11.633577 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.633564    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="360aad55-05a3-4b11-acb5-c15c91f83b49" containerName="maas-api"
Oct 30 18:43:11.633617 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.633586    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ce611c-7b51-4ad0-9577-c9d1c1630836" containerName="maas-api"
Oct 30 18:43:11.638736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.638700    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.645656 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.645606    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-7a8fa2-openshift-default-dockercfg-dddfr\""
Oct 30 18:43:11.653377 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.653338    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"]
Oct 30 18:43:11.732735 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.732685    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"]
Oct 30 18:43:11.748232 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.748174    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9aded813-598bfc4c-k2t5v"]
Oct 30 18:43:11.776652 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.767611    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77c7h\" (UniqueName: \"kubernetes.io/projected/72834257-61fb-4596-8b82-5917990f7469-kube-api-access-77c7h\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.776652 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.768538    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.776652 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.768956    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72834257-61fb-4596-8b82-5917990f7469-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.776652 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.775491    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72834257-61fb-4596-8b82-5917990f7469-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.776652 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.775876    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.776652 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.775906    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.776652 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.775977    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.776652 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.776073    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.776652 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.776175    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72834257-61fb-4596-8b82-5917990f7469-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.876994 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.876942    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72834257-61fb-4596-8b82-5917990f7469-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.877277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.877006    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.877277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.877040    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.877277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.877083    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.877277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.877183    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.877277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.877218    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72834257-61fb-4596-8b82-5917990f7469-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.877277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.877267    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77c7h\" (UniqueName: \"kubernetes.io/projected/72834257-61fb-4596-8b82-5917990f7469-kube-api-access-77c7h\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.877487 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.877296    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.877487 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.877329    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72834257-61fb-4596-8b82-5917990f7469-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.877581 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.877548    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.877581 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.877566    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.877669 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.877632    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.877867 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.877846    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.878240 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.878208    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72834257-61fb-4596-8b82-5917990f7469-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.880012 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.879988    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.880405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.880381    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72834257-61fb-4596-8b82-5917990f7469-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.894911 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.894868    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72834257-61fb-4596-8b82-5917990f7469-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.895103 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.894958    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77c7h\" (UniqueName: \"kubernetes.io/projected/72834257-61fb-4596-8b82-5917990f7469-kube-api-access-77c7h\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:11.956542 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:11.956487    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:12.249983 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:12.249944    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"]
Oct 30 18:43:12.251826 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:43:12.251783    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72834257_61fb_4596_8b82_5917990f7469.slice/crio-7787357855e13b7e77bc2b02d1b17b9f45749377984dd9f371c8bd12f3b89a4c WatchSource:0}: Error finding container 7787357855e13b7e77bc2b02d1b17b9f45749377984dd9f371c8bd12f3b89a4c: Status 404 returned error can't find the container with id 7787357855e13b7e77bc2b02d1b17b9f45749377984dd9f371c8bd12f3b89a4c
Oct 30 18:43:12.254647 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:12.254601    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:12.254744 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:12.254693    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:12.254744 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:12.254721    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:12.265056 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:12.265014    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15ce611c-7b51-4ad0-9577-c9d1c1630836" path="/var/lib/kubelet/pods/15ce611c-7b51-4ad0-9577-c9d1c1630836/volumes"
Oct 30 18:43:12.265562 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:12.265545    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="360aad55-05a3-4b11-acb5-c15c91f83b49" path="/var/lib/kubelet/pods/360aad55-05a3-4b11-acb5-c15c91f83b49/volumes"
Oct 30 18:43:12.418615 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:12.418563    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" event={"ID":"72834257-61fb-4596-8b82-5917990f7469","Type":"ContainerStarted","Data":"af5ea31094e675d08611808658b96313d4e5fcd8dd305bd3c838b87a8ba746fc"}
Oct 30 18:43:12.419166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:12.418634    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" event={"ID":"72834257-61fb-4596-8b82-5917990f7469","Type":"ContainerStarted","Data":"7787357855e13b7e77bc2b02d1b17b9f45749377984dd9f371c8bd12f3b89a4c"}
Oct 30 18:43:12.444550 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:12.444474    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" podStartSLOduration=1.444453171 podStartE2EDuration="1.444453171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:12.443055537 +0000 UTC m=+606.311473866" watchObservedRunningTime="2026-10-30 18:43:12.444453171 +0000 UTC m=+606.312871501"
Oct 30 18:43:12.855448 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:12.855401    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:43:12.957094 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:12.957047    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:13.964397 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:13.964350    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:14.308905 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.308785    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"]
Oct 30 18:43:14.313769 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.313740    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.321616 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.321581    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-da70a5-openshift-default-dockercfg-vx9wq\""
Oct 30 18:43:14.344762 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.342104    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"]
Oct 30 18:43:14.345801 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.345693    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7" podUID="8a92661a-02c0-4788-8f95-0fbd601cabe8" containerName="payload-pre-processing" containerID="cri-o://fa805a16fa85d8cf04b8cbe7aa36c5d899da3ffa2baa28cc4d49e5aaef7e5a89" gracePeriod=25
Oct 30 18:43:14.407256 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.407209    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.407566 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.407371    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dwbm\" (UniqueName: \"kubernetes.io/projected/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-kube-api-access-8dwbm\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.407566 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.407441    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.407566 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.407507    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.407758 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.407565    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.407758 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.407705    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.407758 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.407749    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.407923 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.407902    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.407980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.407927    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.471166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.469974    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:14.471673 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.471606    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:14.529310 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.527275    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.529310 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.528281    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.529310 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.528393    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.529310 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.528439    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.529310 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.528476    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.529853 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.529480    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.529853 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.529535    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.529853 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.529577    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.529853 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.529690    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dwbm\" (UniqueName: \"kubernetes.io/projected/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-kube-api-access-8dwbm\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.530347 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.530314    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.531413 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.531358    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.536076 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.536019    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.536381 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.536101    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.536840 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.536819    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.537890 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.537847    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.556203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.552396    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.556767 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.556657    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.558344 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.558293    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dwbm\" (UniqueName: \"kubernetes.io/projected/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-kube-api-access-8dwbm\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.622866 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.622741    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd" podUID="424cc5a4-a1f7-4d36-ae31-497606a79c16" containerName="payload-processing" containerID="cri-o://9f778f10ff9ab0ed2d667b47e86a900909a19348ef73f76d657dd427812c13b8" gracePeriod=25
Oct 30 18:43:14.632015 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.631858    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:14.645442 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.645075    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p" podUID="26724afc-c0be-4771-a468-1ab2d4d08e52" containerName="payload-pre-processing" containerID="cri-o://633b154f867d42b92341c3ed2111408412983157855e50788cee2605cdf8313a" gracePeriod=25
Oct 30 18:43:14.764839 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.764796    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"
Oct 30 18:43:14.836181 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.833758    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a92661a-02c0-4788-8f95-0fbd601cabe8-kube-api-access-528mv\" (UniqueName: \"kubernetes.io/projected/8a92661a-02c0-4788-8f95-0fbd601cabe8-kube-api-access-528mv\") pod \"8a92661a-02c0-4788-8f95-0fbd601cabe8\" (UID: \"8a92661a-02c0-4788-8f95-0fbd601cabe8\") "
Oct 30 18:43:14.836181 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.833997    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a92661a-02c0-4788-8f95-0fbd601cabe8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a92661a-02c0-4788-8f95-0fbd601cabe8-plugins-config-volume\") pod \"8a92661a-02c0-4788-8f95-0fbd601cabe8\" (UID: \"8a92661a-02c0-4788-8f95-0fbd601cabe8\") "
Oct 30 18:43:14.836181 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.835376    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a92661a-02c0-4788-8f95-0fbd601cabe8-plugins-config-volume" pod "8a92661a-02c0-4788-8f95-0fbd601cabe8" (UID: "8a92661a-02c0-4788-8f95-0fbd601cabe8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:14.840085 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.840008    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a92661a-02c0-4788-8f95-0fbd601cabe8-kube-api-access-528mv" pod "8a92661a-02c0-4788-8f95-0fbd601cabe8" (UID: "8a92661a-02c0-4788-8f95-0fbd601cabe8"). InnerVolumeSpecName "kube-api-access-528mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:14.935421 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.935382    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a92661a-02c0-4788-8f95-0fbd601cabe8-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:14.935421 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.935415    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-528mv\" (UniqueName: \"kubernetes.io/projected/8a92661a-02c0-4788-8f95-0fbd601cabe8-kube-api-access-528mv\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:14.946569 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.946534    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"]
Oct 30 18:43:14.950098 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:43:14.950044    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a5e1e6_da62_4ccc_853e_d7346ac1ad5b.slice/crio-7bad246c063b7f1717d7df11125c5f4c3eb8734ba37b1c54b4e818c7180c3385 WatchSource:0}: Error finding container 7bad246c063b7f1717d7df11125c5f4c3eb8734ba37b1c54b4e818c7180c3385: Status 404 returned error can't find the container with id 7bad246c063b7f1717d7df11125c5f4c3eb8734ba37b1c54b4e818c7180c3385
Oct 30 18:43:14.954015 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.953947    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:14.954181 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.954090    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:14.954181 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:14.954165    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:15.056358 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.056316    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"
Oct 30 18:43:15.060097 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.060069    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"
Oct 30 18:43:15.137039 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.137001    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26724afc-c0be-4771-a468-1ab2d4d08e52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26724afc-c0be-4771-a468-1ab2d4d08e52-plugins-config-volume\") pod \"26724afc-c0be-4771-a468-1ab2d4d08e52\" (UID: \"26724afc-c0be-4771-a468-1ab2d4d08e52\") "
Oct 30 18:43:15.137300 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.137103    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424cc5a4-a1f7-4d36-ae31-497606a79c16-kube-api-access-6zz6x\" (UniqueName: \"kubernetes.io/projected/424cc5a4-a1f7-4d36-ae31-497606a79c16-kube-api-access-6zz6x\") pod \"424cc5a4-a1f7-4d36-ae31-497606a79c16\" (UID: \"424cc5a4-a1f7-4d36-ae31-497606a79c16\") "
Oct 30 18:43:15.137300 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.137201    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26724afc-c0be-4771-a468-1ab2d4d08e52-kube-api-access-jtptl\" (UniqueName: \"kubernetes.io/projected/26724afc-c0be-4771-a468-1ab2d4d08e52-kube-api-access-jtptl\") pod \"26724afc-c0be-4771-a468-1ab2d4d08e52\" (UID: \"26724afc-c0be-4771-a468-1ab2d4d08e52\") "
Oct 30 18:43:15.137428 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.137303    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424cc5a4-a1f7-4d36-ae31-497606a79c16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424cc5a4-a1f7-4d36-ae31-497606a79c16-plugins-config-volume\") pod \"424cc5a4-a1f7-4d36-ae31-497606a79c16\" (UID: \"424cc5a4-a1f7-4d36-ae31-497606a79c16\") "
Oct 30 18:43:15.137660 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.137620    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26724afc-c0be-4771-a468-1ab2d4d08e52-plugins-config-volume" pod "26724afc-c0be-4771-a468-1ab2d4d08e52" (UID: "26724afc-c0be-4771-a468-1ab2d4d08e52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:15.137738 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.137710    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424cc5a4-a1f7-4d36-ae31-497606a79c16-plugins-config-volume" pod "424cc5a4-a1f7-4d36-ae31-497606a79c16" (UID: "424cc5a4-a1f7-4d36-ae31-497606a79c16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:15.137818 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.137799    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26724afc-c0be-4771-a468-1ab2d4d08e52-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:15.140008 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.139953    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424cc5a4-a1f7-4d36-ae31-497606a79c16-kube-api-access-6zz6x" pod "424cc5a4-a1f7-4d36-ae31-497606a79c16" (UID: "424cc5a4-a1f7-4d36-ae31-497606a79c16"). InnerVolumeSpecName "kube-api-access-6zz6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:15.140395 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.140372    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26724afc-c0be-4771-a468-1ab2d4d08e52-kube-api-access-jtptl" pod "26724afc-c0be-4771-a468-1ab2d4d08e52" (UID: "26724afc-c0be-4771-a468-1ab2d4d08e52"). InnerVolumeSpecName "kube-api-access-jtptl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:15.239405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.239253    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zz6x\" (UniqueName: \"kubernetes.io/projected/424cc5a4-a1f7-4d36-ae31-497606a79c16-kube-api-access-6zz6x\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:15.239405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.239320    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtptl\" (UniqueName: \"kubernetes.io/projected/26724afc-c0be-4771-a468-1ab2d4d08e52-kube-api-access-jtptl\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:15.239405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.239337    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424cc5a4-a1f7-4d36-ae31-497606a79c16-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:15.430213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.429352    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"]
Oct 30 18:43:15.430213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.429664    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp" podUID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" containerName="istio-proxy" containerID="cri-o://a87d81511bfa54c369c0f7d0a984493bed2e8b54702e5288c9979deb2b8fb084" gracePeriod=30
Oct 30 18:43:15.480118 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.480071    4372 generic.go:356] "Generic (PLEG): container finished" podID="2b704d30-26f4-4f7f-9b78-aece8de84de0" containerID="8aa4c1004dd7408374ee147e3c9bc8c068e8ac5482a28f761f251000aabc9d31" exitCode=0
Oct 30 18:43:15.480390 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.480164    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2" event={"ID":"2b704d30-26f4-4f7f-9b78-aece8de84de0","Type":"ContainerDied","Data":"8aa4c1004dd7408374ee147e3c9bc8c068e8ac5482a28f761f251000aabc9d31"}
Oct 30 18:43:15.480390 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.480264    4372 scope.go:122] "RemoveContainer" containerID="9905403da43802713fc5bae049dd0ab6aa3a5754ca416ff3be96ef421ace27f2"
Oct 30 18:43:15.482037 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.481988    4372 generic.go:356] "Generic (PLEG): container finished" podID="8a92661a-02c0-4788-8f95-0fbd601cabe8" containerID="fa805a16fa85d8cf04b8cbe7aa36c5d899da3ffa2baa28cc4d49e5aaef7e5a89" exitCode=0
Oct 30 18:43:15.482282 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.482082    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"
Oct 30 18:43:15.482282 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.482091    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7" event={"ID":"8a92661a-02c0-4788-8f95-0fbd601cabe8","Type":"ContainerDied","Data":"fa805a16fa85d8cf04b8cbe7aa36c5d899da3ffa2baa28cc4d49e5aaef7e5a89"}
Oct 30 18:43:15.482282 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.482186    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7" event={"ID":"8a92661a-02c0-4788-8f95-0fbd601cabe8","Type":"ContainerDied","Data":"78c9dd1e30521b05d90fc0e621c39c213d433dc0209137e2c575f2155fb79578"}
Oct 30 18:43:15.492505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.485241    4372 generic.go:356] "Generic (PLEG): container finished" podID="26724afc-c0be-4771-a468-1ab2d4d08e52" containerID="633b154f867d42b92341c3ed2111408412983157855e50788cee2605cdf8313a" exitCode=0
Oct 30 18:43:15.492505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.485317    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"
Oct 30 18:43:15.492505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.485415    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p" event={"ID":"26724afc-c0be-4771-a468-1ab2d4d08e52","Type":"ContainerDied","Data":"633b154f867d42b92341c3ed2111408412983157855e50788cee2605cdf8313a"}
Oct 30 18:43:15.492505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.485461    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p" event={"ID":"26724afc-c0be-4771-a468-1ab2d4d08e52","Type":"ContainerDied","Data":"fd6526f7b85f91130cf2d0ac736ec2928c26d9d3ad9f9f7b747ab37f1b70bdf7"}
Oct 30 18:43:15.492505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.488827    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw" event={"ID":"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b","Type":"ContainerStarted","Data":"cc880ea57511314e5161ed64b11449474d54fe1e325a8c25bd0e81a3f734230c"}
Oct 30 18:43:15.492505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.488897    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw" event={"ID":"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b","Type":"ContainerStarted","Data":"7bad246c063b7f1717d7df11125c5f4c3eb8734ba37b1c54b4e818c7180c3385"}
Oct 30 18:43:15.492505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.491594    4372 generic.go:356] "Generic (PLEG): container finished" podID="424cc5a4-a1f7-4d36-ae31-497606a79c16" containerID="9f778f10ff9ab0ed2d667b47e86a900909a19348ef73f76d657dd427812c13b8" exitCode=0
Oct 30 18:43:15.492505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.491722    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"
Oct 30 18:43:15.492505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.491774    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd" event={"ID":"424cc5a4-a1f7-4d36-ae31-497606a79c16","Type":"ContainerDied","Data":"9f778f10ff9ab0ed2d667b47e86a900909a19348ef73f76d657dd427812c13b8"}
Oct 30 18:43:15.492505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.491835    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd" event={"ID":"424cc5a4-a1f7-4d36-ae31-497606a79c16","Type":"ContainerDied","Data":"bc8bdea1a83b0514785966873ddf945833662012ee6f9b081c68a96a50ccee00"}
Oct 30 18:43:15.501261 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.500679    4372 scope.go:122] "RemoveContainer" containerID="fa805a16fa85d8cf04b8cbe7aa36c5d899da3ffa2baa28cc4d49e5aaef7e5a89"
Oct 30 18:43:15.543490 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.537404    4372 scope.go:122] "RemoveContainer" containerID="fa805a16fa85d8cf04b8cbe7aa36c5d899da3ffa2baa28cc4d49e5aaef7e5a89"
Oct 30 18:43:15.575560 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:43:15.569810    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa805a16fa85d8cf04b8cbe7aa36c5d899da3ffa2baa28cc4d49e5aaef7e5a89\": container with ID starting with fa805a16fa85d8cf04b8cbe7aa36c5d899da3ffa2baa28cc4d49e5aaef7e5a89 not found: ID does not exist" containerID="fa805a16fa85d8cf04b8cbe7aa36c5d899da3ffa2baa28cc4d49e5aaef7e5a89"
Oct 30 18:43:15.575560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.569875    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa805a16fa85d8cf04b8cbe7aa36c5d899da3ffa2baa28cc4d49e5aaef7e5a89"} err="failed to get container status \"fa805a16fa85d8cf04b8cbe7aa36c5d899da3ffa2baa28cc4d49e5aaef7e5a89\": rpc error: code = NotFound desc = could not find container \"fa805a16fa85d8cf04b8cbe7aa36c5d899da3ffa2baa28cc4d49e5aaef7e5a89\": container with ID starting with fa805a16fa85d8cf04b8cbe7aa36c5d899da3ffa2baa28cc4d49e5aaef7e5a89 not found: ID does not exist"
Oct 30 18:43:15.575560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.569914    4372 scope.go:122] "RemoveContainer" containerID="633b154f867d42b92341c3ed2111408412983157855e50788cee2605cdf8313a"
Oct 30 18:43:15.583427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.583318    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"]
Oct 30 18:43:15.586750 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.586519    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7289525-5766bdb8cf-sdsk7"]
Oct 30 18:43:15.597652 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.597609    4372 scope.go:122] "RemoveContainer" containerID="633b154f867d42b92341c3ed2111408412983157855e50788cee2605cdf8313a"
Oct 30 18:43:15.599018 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:43:15.598323    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"633b154f867d42b92341c3ed2111408412983157855e50788cee2605cdf8313a\": container with ID starting with 633b154f867d42b92341c3ed2111408412983157855e50788cee2605cdf8313a not found: ID does not exist" containerID="633b154f867d42b92341c3ed2111408412983157855e50788cee2605cdf8313a"
Oct 30 18:43:15.599018 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.598613    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633b154f867d42b92341c3ed2111408412983157855e50788cee2605cdf8313a"} err="failed to get container status \"633b154f867d42b92341c3ed2111408412983157855e50788cee2605cdf8313a\": rpc error: code = NotFound desc = could not find container \"633b154f867d42b92341c3ed2111408412983157855e50788cee2605cdf8313a\": container with ID starting with 633b154f867d42b92341c3ed2111408412983157855e50788cee2605cdf8313a not found: ID does not exist"
Oct 30 18:43:15.599018 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.598649    4372 scope.go:122] "RemoveContainer" containerID="9f778f10ff9ab0ed2d667b47e86a900909a19348ef73f76d657dd427812c13b8"
Oct 30 18:43:15.617210 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.617139    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw" podStartSLOduration=1.6171088390000001 podStartE2EDuration="1.617108839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:15.614627408 +0000 UTC m=+609.483045739" watchObservedRunningTime="2026-10-30 18:43:15.617108839 +0000 UTC m=+609.485527169"
Oct 30 18:43:15.627679 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.627565    4372 scope.go:122] "RemoveContainer" containerID="9f778f10ff9ab0ed2d667b47e86a900909a19348ef73f76d657dd427812c13b8"
Oct 30 18:43:15.627894 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:43:15.627871    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f778f10ff9ab0ed2d667b47e86a900909a19348ef73f76d657dd427812c13b8\": container with ID starting with 9f778f10ff9ab0ed2d667b47e86a900909a19348ef73f76d657dd427812c13b8 not found: ID does not exist" containerID="9f778f10ff9ab0ed2d667b47e86a900909a19348ef73f76d657dd427812c13b8"
Oct 30 18:43:15.627935 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.627904    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f778f10ff9ab0ed2d667b47e86a900909a19348ef73f76d657dd427812c13b8"} err="failed to get container status \"9f778f10ff9ab0ed2d667b47e86a900909a19348ef73f76d657dd427812c13b8\": rpc error: code = NotFound desc = could not find container \"9f778f10ff9ab0ed2d667b47e86a900909a19348ef73f76d657dd427812c13b8\": container with ID starting with 9f778f10ff9ab0ed2d667b47e86a900909a19348ef73f76d657dd427812c13b8 not found: ID does not exist"
Oct 30 18:43:15.631385 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.631350    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:43:15.633058 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.633029    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:15.637025 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.636979    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"]
Oct 30 18:43:15.644918 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.644872    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-78934afe-59db46b44b-9wnpd"]
Oct 30 18:43:15.656015 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.655970    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-podinfo\") pod \"2b704d30-26f4-4f7f-9b78-aece8de84de0\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") "
Oct 30 18:43:15.656327 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.656033    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b704d30-26f4-4f7f-9b78-aece8de84de0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b704d30-26f4-4f7f-9b78-aece8de84de0-istiod-ca-cert\") pod \"2b704d30-26f4-4f7f-9b78-aece8de84de0\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") "
Oct 30 18:43:15.656327 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.656241    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-envoy\") pod \"2b704d30-26f4-4f7f-9b78-aece8de84de0\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") "
Oct 30 18:43:15.657540 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.656328    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b704d30-26f4-4f7f-9b78-aece8de84de0-kube-api-access-nlpxd\" (UniqueName: \"kubernetes.io/projected/2b704d30-26f4-4f7f-9b78-aece8de84de0-kube-api-access-nlpxd\") pod \"2b704d30-26f4-4f7f-9b78-aece8de84de0\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") "
Oct 30 18:43:15.657540 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.656396    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-credential-socket\") pod \"2b704d30-26f4-4f7f-9b78-aece8de84de0\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") "
Oct 30 18:43:15.657540 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.656481    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-workload-certs\") pod \"2b704d30-26f4-4f7f-9b78-aece8de84de0\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") "
Oct 30 18:43:15.657540 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.656555    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-token\" (UniqueName: \"kubernetes.io/projected/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-token\") pod \"2b704d30-26f4-4f7f-9b78-aece8de84de0\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") "
Oct 30 18:43:15.657540 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.656591    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-data\") pod \"2b704d30-26f4-4f7f-9b78-aece8de84de0\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") "
Oct 30 18:43:15.657540 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.656646    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-workload-socket\") pod \"2b704d30-26f4-4f7f-9b78-aece8de84de0\" (UID: \"2b704d30-26f4-4f7f-9b78-aece8de84de0\") "
Oct 30 18:43:15.657540 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.656751    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b704d30-26f4-4f7f-9b78-aece8de84de0-istiod-ca-cert" pod "2b704d30-26f4-4f7f-9b78-aece8de84de0" (UID: "2b704d30-26f4-4f7f-9b78-aece8de84de0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:15.657540 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.657160    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-workload-certs" pod "2b704d30-26f4-4f7f-9b78-aece8de84de0" (UID: "2b704d30-26f4-4f7f-9b78-aece8de84de0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:15.657540 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.657503    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-credential-socket" pod "2b704d30-26f4-4f7f-9b78-aece8de84de0" (UID: "2b704d30-26f4-4f7f-9b78-aece8de84de0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:15.658097 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.657786    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-data" pod "2b704d30-26f4-4f7f-9b78-aece8de84de0" (UID: "2b704d30-26f4-4f7f-9b78-aece8de84de0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:15.658529 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.658500    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b704d30-26f4-4f7f-9b78-aece8de84de0-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:15.658675 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.658531    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:15.658675 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.658547    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:15.658675 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.658560    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:15.658675 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.658563    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-workload-socket" pod "2b704d30-26f4-4f7f-9b78-aece8de84de0" (UID: "2b704d30-26f4-4f7f-9b78-aece8de84de0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:15.659887 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.659853    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-podinfo" pod "2b704d30-26f4-4f7f-9b78-aece8de84de0" (UID: "2b704d30-26f4-4f7f-9b78-aece8de84de0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:43:15.660308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.660278    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b704d30-26f4-4f7f-9b78-aece8de84de0-kube-api-access-nlpxd" pod "2b704d30-26f4-4f7f-9b78-aece8de84de0" (UID: "2b704d30-26f4-4f7f-9b78-aece8de84de0"). InnerVolumeSpecName "kube-api-access-nlpxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:15.660308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.660289    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-token" pod "2b704d30-26f4-4f7f-9b78-aece8de84de0" (UID: "2b704d30-26f4-4f7f-9b78-aece8de84de0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:15.660598 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.660577    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"]
Oct 30 18:43:15.660685 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.660664    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-envoy" pod "2b704d30-26f4-4f7f-9b78-aece8de84de0" (UID: "2b704d30-26f4-4f7f-9b78-aece8de84de0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:15.668655 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.668625    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78934afe-868fcc9cff-ms82p"]
Oct 30 18:43:15.760240 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.760091    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:15.760240 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.760157    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlpxd\" (UniqueName: \"kubernetes.io/projected/2b704d30-26f4-4f7f-9b78-aece8de84de0-kube-api-access-nlpxd\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:15.760240 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.760177    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:15.760240 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.760190    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b704d30-26f4-4f7f-9b78-aece8de84de0-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:15.760240 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:15.760203    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b704d30-26f4-4f7f-9b78-aece8de84de0-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:16.266258 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.266216    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26724afc-c0be-4771-a468-1ab2d4d08e52" path="/var/lib/kubelet/pods/26724afc-c0be-4771-a468-1ab2d4d08e52/volumes"
Oct 30 18:43:16.266820 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.266759    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424cc5a4-a1f7-4d36-ae31-497606a79c16" path="/var/lib/kubelet/pods/424cc5a4-a1f7-4d36-ae31-497606a79c16/volumes"
Oct 30 18:43:16.267238 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.267216    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a92661a-02c0-4788-8f95-0fbd601cabe8" path="/var/lib/kubelet/pods/8a92661a-02c0-4788-8f95-0fbd601cabe8/volumes"
Oct 30 18:43:16.422672 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.422596    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"]
Oct 30 18:43:16.423103 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.423087    4372 cpu_manager.go:414] "containerMap: removing container" podUID="424cc5a4-a1f7-4d36-ae31-497606a79c16" containerName="payload-processing"
Oct 30 18:43:16.423213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.423104    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424cc5a4-a1f7-4d36-ae31-497606a79c16" containerName="payload-processing"
Oct 30 18:43:16.423213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.423139    4372 cpu_manager.go:414] "containerMap: removing container" podUID="26724afc-c0be-4771-a468-1ab2d4d08e52" containerName="payload-pre-processing"
Oct 30 18:43:16.423213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.423145    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26724afc-c0be-4771-a468-1ab2d4d08e52" containerName="payload-pre-processing"
Oct 30 18:43:16.423213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.423160    4372 cpu_manager.go:414] "containerMap: removing container" podUID="8a92661a-02c0-4788-8f95-0fbd601cabe8" containerName="payload-pre-processing"
Oct 30 18:43:16.423213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.423166    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a92661a-02c0-4788-8f95-0fbd601cabe8" containerName="payload-pre-processing"
Oct 30 18:43:16.423213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.423185    4372 cpu_manager.go:414] "containerMap: removing container" podUID="2b704d30-26f4-4f7f-9b78-aece8de84de0" containerName="istio-proxy"
Oct 30 18:43:16.423213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.423190    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b704d30-26f4-4f7f-9b78-aece8de84de0" containerName="istio-proxy"
Oct 30 18:43:16.423450 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.423266    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="26724afc-c0be-4771-a468-1ab2d4d08e52" containerName="payload-pre-processing"
Oct 30 18:43:16.423450 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.423278    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a92661a-02c0-4788-8f95-0fbd601cabe8" containerName="payload-pre-processing"
Oct 30 18:43:16.423450 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.423287    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b704d30-26f4-4f7f-9b78-aece8de84de0" containerName="istio-proxy"
Oct 30 18:43:16.423450 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.423297    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="424cc5a4-a1f7-4d36-ae31-497606a79c16" containerName="payload-processing"
Oct 30 18:43:16.428348 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.428306    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"
Oct 30 18:43:16.438863 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.438831    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"]
Oct 30 18:43:16.468392 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.468331    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89c0e0e-7b58-4c7f-a959-75e56c711f6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj\" (UID: \"d89c0e0e-7b58-4c7f-a959-75e56c711f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"
Oct 30 18:43:16.468615 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.468516    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5rl\" (UniqueName: \"kubernetes.io/projected/d89c0e0e-7b58-4c7f-a959-75e56c711f6f-kube-api-access-ww5rl\") pod \"payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj\" (UID: \"d89c0e0e-7b58-4c7f-a959-75e56c711f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"
Oct 30 18:43:16.524255 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.512665    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-7a8fa2\""
Oct 30 18:43:16.534416 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.532617    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2" event={"ID":"2b704d30-26f4-4f7f-9b78-aece8de84de0","Type":"ContainerDied","Data":"0d3dcddd3b2e4e39dba90039c8f44372cff5f483a369122ce0aa0adfb37daf6a"}
Oct 30 18:43:16.534416 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.532703    4372 scope.go:122] "RemoveContainer" containerID="8aa4c1004dd7408374ee147e3c9bc8c068e8ac5482a28f761f251000aabc9d31"
Oct 30 18:43:16.534416 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.533042    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"
Oct 30 18:43:16.548695 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.541064    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-7a8fa2-dockercfg-bjbpb\""
Oct 30 18:43:16.571960 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.569945    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5rl\" (UniqueName: \"kubernetes.io/projected/d89c0e0e-7b58-4c7f-a959-75e56c711f6f-kube-api-access-ww5rl\") pod \"payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj\" (UID: \"d89c0e0e-7b58-4c7f-a959-75e56c711f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"
Oct 30 18:43:16.571960 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.570295    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89c0e0e-7b58-4c7f-a959-75e56c711f6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj\" (UID: \"d89c0e0e-7b58-4c7f-a959-75e56c711f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"
Oct 30 18:43:16.572360 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.572317    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89c0e0e-7b58-4c7f-a959-75e56c711f6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj\" (UID: \"d89c0e0e-7b58-4c7f-a959-75e56c711f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"
Oct 30 18:43:16.580788 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.580724    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"]
Oct 30 18:43:16.589118 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.589075    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5rl\" (UniqueName: \"kubernetes.io/projected/d89c0e0e-7b58-4c7f-a959-75e56c711f6f-kube-api-access-ww5rl\") pod \"payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj\" (UID: \"d89c0e0e-7b58-4c7f-a959-75e56c711f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"
Oct 30 18:43:16.591611 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.591563    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-78934afe-gw-openshift-default-69cbb855b9-7wbl2"]
Oct 30 18:43:16.620268 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.620218    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"]
Oct 30 18:43:16.620800 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.620780    4372 cpu_manager.go:414] "containerMap: removing container" podUID="2b704d30-26f4-4f7f-9b78-aece8de84de0" containerName="istio-proxy"
Oct 30 18:43:16.620916 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.620803    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b704d30-26f4-4f7f-9b78-aece8de84de0" containerName="istio-proxy"
Oct 30 18:43:16.620916 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.620898    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b704d30-26f4-4f7f-9b78-aece8de84de0" containerName="istio-proxy"
Oct 30 18:43:16.625806 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.625775    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.641392 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.641352    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:16.643009 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.642930    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"]
Oct 30 18:43:16.671153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.671081    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10111dcc-0436-4206-b798-d0bac119328a-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.671425 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.671269    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxzxj\" (UniqueName: \"kubernetes.io/projected/10111dcc-0436-4206-b798-d0bac119328a-kube-api-access-hxzxj\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.671425 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.671325    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.671425 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.671386    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.671574 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.671442    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10111dcc-0436-4206-b798-d0bac119328a-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.671574 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.671473    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.671574 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.671493    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.671574 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.671518    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10111dcc-0436-4206-b798-d0bac119328a-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.671705 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.671642    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.726825 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.726701    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8a92cc-openshift-default-dockercfg-vj746\""
Oct 30 18:43:16.748763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.747365    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"
Oct 30 18:43:16.795062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.788733    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.795062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.788842    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10111dcc-0436-4206-b798-d0bac119328a-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.795062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.788892    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.795062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.788952    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10111dcc-0436-4206-b798-d0bac119328a-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.795062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.789062    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxzxj\" (UniqueName: \"kubernetes.io/projected/10111dcc-0436-4206-b798-d0bac119328a-kube-api-access-hxzxj\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.795062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.789200    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.795062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.789257    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.795062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.789321    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10111dcc-0436-4206-b798-d0bac119328a-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.795062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.789373    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.795062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.790027    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.795062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.791431    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.795062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.792274    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.795062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.792881    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10111dcc-0436-4206-b798-d0bac119328a-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.795062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.794875    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.796593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.796562    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10111dcc-0436-4206-b798-d0bac119328a-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.797006 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.796930    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.807206 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.807163    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10111dcc-0436-4206-b798-d0bac119328a-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.807422 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.807315    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxzxj\" (UniqueName: \"kubernetes.io/projected/10111dcc-0436-4206-b798-d0bac119328a-kube-api-access-hxzxj\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:16.947198 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:16.947160    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:17.113575 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.113539    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"]
Oct 30 18:43:17.116287 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:43:17.116254    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89c0e0e_7b58_4c7f_a959_75e56c711f6f.slice/crio-df299f5aba28ee387904ba4cdcfc3d5c4a6db5890449e2d1549ad3fe99b99f79 WatchSource:0}: Error finding container df299f5aba28ee387904ba4cdcfc3d5c4a6db5890449e2d1549ad3fe99b99f79: Status 404 returned error can't find the container with id df299f5aba28ee387904ba4cdcfc3d5c4a6db5890449e2d1549ad3fe99b99f79
Oct 30 18:43:17.287736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.287706    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"]
Oct 30 18:43:17.290513 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:43:17.290468    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10111dcc_0436_4206_b798_d0bac119328a.slice/crio-2b7804859edc34d9dda5af3c37312e1783735d9e34b16e3ecdddd86606621e31 WatchSource:0}: Error finding container 2b7804859edc34d9dda5af3c37312e1783735d9e34b16e3ecdddd86606621e31: Status 404 returned error can't find the container with id 2b7804859edc34d9dda5af3c37312e1783735d9e34b16e3ecdddd86606621e31
Oct 30 18:43:17.293647 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.293595    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:17.293781 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.293693    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:17.293781 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.293729    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:17.319166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.319078    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:43:17.558097 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.558050    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj" event={"ID":"d89c0e0e-7b58-4c7f-a959-75e56c711f6f","Type":"ContainerStarted","Data":"4f045bda6f80c4206ddb7ebd55253bc7b05e80d74f794d12d0d4f8c3328bfaa6"}
Oct 30 18:43:17.558355 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.558165    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj" event={"ID":"d89c0e0e-7b58-4c7f-a959-75e56c711f6f","Type":"ContainerStarted","Data":"df299f5aba28ee387904ba4cdcfc3d5c4a6db5890449e2d1549ad3fe99b99f79"}
Oct 30 18:43:17.558355 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.558294    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"
Oct 30 18:43:17.561387 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.561339    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" event={"ID":"10111dcc-0436-4206-b798-d0bac119328a","Type":"ContainerStarted","Data":"704cc894c079ca154a3adda2c659990411c5e2ad0a197eccc1026637ff020c3e"}
Oct 30 18:43:17.561612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.561418    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" event={"ID":"10111dcc-0436-4206-b798-d0bac119328a","Type":"ContainerStarted","Data":"2b7804859edc34d9dda5af3c37312e1783735d9e34b16e3ecdddd86606621e31"}
Oct 30 18:43:17.561612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.561458    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:17.563262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.563232    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:17.586061 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.585935    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj" podStartSLOduration=1.58591688 podStartE2EDuration="1.58591688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:17.583934013 +0000 UTC m=+611.452352344" watchObservedRunningTime="2026-10-30 18:43:17.58591688 +0000 UTC m=+611.454335209"
Oct 30 18:43:17.615865 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.615807    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" podStartSLOduration=1.615788857 podStartE2EDuration="1.615788857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:17.615665877 +0000 UTC m=+611.484084209" watchObservedRunningTime="2026-10-30 18:43:17.615788857 +0000 UTC m=+611.484207187"
Oct 30 18:43:17.948013 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:17.947918    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:18.267187 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:18.267045    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b704d30-26f4-4f7f-9b78-aece8de84de0" path="/var/lib/kubelet/pods/2b704d30-26f4-4f7f-9b78-aece8de84de0/volumes"
Oct 30 18:43:18.956676 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:18.954933    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:19.601491 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:19.601442    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:19.603086 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:19.603054    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:20.663950 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.663903    4372 generic.go:356] "Generic (PLEG): container finished" podID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" containerID="a87d81511bfa54c369c0f7d0a984493bed2e8b54702e5288c9979deb2b8fb084" exitCode=0
Oct 30 18:43:20.664600 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.663975    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp" event={"ID":"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d","Type":"ContainerDied","Data":"a87d81511bfa54c369c0f7d0a984493bed2e8b54702e5288c9979deb2b8fb084"}
Oct 30 18:43:20.664600 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.664105    4372 scope.go:122] "RemoveContainer" containerID="0d2ed977f1b12ea4e522e4ab5d9521c2b10614cfea4eb43cf3f9da6c29215413"
Oct 30 18:43:20.778814 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.778772    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:43:20.870160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.862256    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-workload-certs\") pod \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") "
Oct 30 18:43:20.870160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.862324    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-kube-api-access-7vbgc\" (UniqueName: \"kubernetes.io/projected/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-kube-api-access-7vbgc\") pod \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") "
Oct 30 18:43:20.870160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.862377    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-podinfo\") pod \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") "
Oct 30 18:43:20.870160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.862402    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-workload-socket\") pod \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") "
Oct 30 18:43:20.870160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.862442    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-token\" (UniqueName: \"kubernetes.io/projected/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-token\") pod \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") "
Oct 30 18:43:20.870160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.862476    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-credential-socket\") pod \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") "
Oct 30 18:43:20.870160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.862503    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-data\") pod \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") "
Oct 30 18:43:20.870160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.862539    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istiod-ca-cert\") pod \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") "
Oct 30 18:43:20.870160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.862611    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-envoy\") pod \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\" (UID: \"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d\") "
Oct 30 18:43:20.870160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.865715    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-workload-certs" pod "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" (UID: "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:20.870160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.866161    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-workload-socket" pod "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" (UID: "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:20.873679 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.873626    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-data" pod "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" (UID: "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:20.873976 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.873833    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istiod-ca-cert" pod "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" (UID: "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:43:20.878672 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.878607    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-credential-socket" pod "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" (UID: "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:20.898839 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.898790    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-envoy" pod "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" (UID: "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:20.909190 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.904805    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-kube-api-access-7vbgc" pod "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" (UID: "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d"). InnerVolumeSpecName "kube-api-access-7vbgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:20.909190 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.904818    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-token" pod "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" (UID: "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:20.910312 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.910236    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-podinfo" pod "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" (UID: "1c2f3a26-d0d7-41af-9a12-b020d1a9e82d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:43:20.964269 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.964216    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.964269 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.964263    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vbgc\" (UniqueName: \"kubernetes.io/projected/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-kube-api-access-7vbgc\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.964269 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.964275    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.964269 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.964284    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.964598 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.964295    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.964598 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.964305    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.964598 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.964314    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.964598 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.964322    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:20.964598 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:20.964331    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:21.671409 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.671374    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"
Oct 30 18:43:21.671409 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.671384    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp" event={"ID":"1c2f3a26-d0d7-41af-9a12-b020d1a9e82d","Type":"ContainerDied","Data":"49c8e6078aff098103e1d053b114690d2c0edefd6df6ee8d04ea723d141f5bbc"}
Oct 30 18:43:21.672035 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.671456    4372 scope.go:122] "RemoveContainer" containerID="a87d81511bfa54c369c0f7d0a984493bed2e8b54702e5288c9979deb2b8fb084"
Oct 30 18:43:21.708563 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.708516    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"]
Oct 30 18:43:21.720215 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.720171    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9aded813-openshift-default-7d59d555c9-9kfqp"]
Oct 30 18:43:21.813509 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.813468    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"]
Oct 30 18:43:21.814275 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.814247    4372 cpu_manager.go:414] "containerMap: removing container" podUID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" containerName="istio-proxy"
Oct 30 18:43:21.814458 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.814277    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" containerName="istio-proxy"
Oct 30 18:43:21.814458 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.814305    4372 cpu_manager.go:414] "containerMap: removing container" podUID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" containerName="istio-proxy"
Oct 30 18:43:21.814458 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.814313    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" containerName="istio-proxy"
Oct 30 18:43:21.814458 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.814445    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" containerName="istio-proxy"
Oct 30 18:43:21.814664 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.814461    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" containerName="istio-proxy"
Oct 30 18:43:21.822672 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.822623    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"
Oct 30 18:43:21.830241 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.830190    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"]
Oct 30 18:43:21.832626 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.832566    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-8a92cc-dockercfg-t74dh\""
Oct 30 18:43:21.887257 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.884151    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23924268-e8d8-4506-8581-5c5ae5fc9ddf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t\" (UID: \"23924268-e8d8-4506-8581-5c5ae5fc9ddf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"
Oct 30 18:43:21.887257 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.884302    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xp5z\" (UniqueName: \"kubernetes.io/projected/23924268-e8d8-4506-8581-5c5ae5fc9ddf-kube-api-access-2xp5z\") pod \"payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t\" (UID: \"23924268-e8d8-4506-8581-5c5ae5fc9ddf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"
Oct 30 18:43:21.894591 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:21.894547    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-8a92cc\""
Oct 30 18:43:22.001087 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.000921    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23924268-e8d8-4506-8581-5c5ae5fc9ddf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t\" (UID: \"23924268-e8d8-4506-8581-5c5ae5fc9ddf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"
Oct 30 18:43:22.004172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.003982    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23924268-e8d8-4506-8581-5c5ae5fc9ddf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t\" (UID: \"23924268-e8d8-4506-8581-5c5ae5fc9ddf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"
Oct 30 18:43:22.004706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.004664    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xp5z\" (UniqueName: \"kubernetes.io/projected/23924268-e8d8-4506-8581-5c5ae5fc9ddf-kube-api-access-2xp5z\") pod \"payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t\" (UID: \"23924268-e8d8-4506-8581-5c5ae5fc9ddf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"
Oct 30 18:43:22.022486 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.022455    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xp5z\" (UniqueName: \"kubernetes.io/projected/23924268-e8d8-4506-8581-5c5ae5fc9ddf-kube-api-access-2xp5z\") pod \"payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t\" (UID: \"23924268-e8d8-4506-8581-5c5ae5fc9ddf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"
Oct 30 18:43:22.145491 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.145439    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"
Oct 30 18:43:22.267378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.267285    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c2f3a26-d0d7-41af-9a12-b020d1a9e82d" path="/var/lib/kubelet/pods/1c2f3a26-d0d7-41af-9a12-b020d1a9e82d/volumes"
Oct 30 18:43:22.426426 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.426383    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"]
Oct 30 18:43:22.431704 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.431667    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"
Oct 30 18:43:22.442085 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.441891    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"]
Oct 30 18:43:22.442287 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.442263    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-84c32a\""
Oct 30 18:43:22.463313 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:43:22.463274    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23924268_e8d8_4506_8581_5c5ae5fc9ddf.slice/crio-62a9eb50f75b1642d0e7c3d3ac22c8756c8e131756567dadbc60ad120b144457 WatchSource:0}: Error finding container 62a9eb50f75b1642d0e7c3d3ac22c8756c8e131756567dadbc60ad120b144457: Status 404 returned error can't find the container with id 62a9eb50f75b1642d0e7c3d3ac22c8756c8e131756567dadbc60ad120b144457
Oct 30 18:43:22.464038 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.464005    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"]
Oct 30 18:43:22.465545 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.465518    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-84c32a-dockercfg-wcktw\""
Oct 30 18:43:22.515049 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.515007    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9196fb90-c915-443e-bd49-109a77351d03-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7\" (UID: \"9196fb90-c915-443e-bd49-109a77351d03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"
Oct 30 18:43:22.515315 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.515089    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4lv\" (UniqueName: \"kubernetes.io/projected/9196fb90-c915-443e-bd49-109a77351d03-kube-api-access-8c4lv\") pod \"payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7\" (UID: \"9196fb90-c915-443e-bd49-109a77351d03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"
Oct 30 18:43:22.620484 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.620402    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9196fb90-c915-443e-bd49-109a77351d03-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7\" (UID: \"9196fb90-c915-443e-bd49-109a77351d03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"
Oct 30 18:43:22.620711 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.620545    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4lv\" (UniqueName: \"kubernetes.io/projected/9196fb90-c915-443e-bd49-109a77351d03-kube-api-access-8c4lv\") pod \"payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7\" (UID: \"9196fb90-c915-443e-bd49-109a77351d03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"
Oct 30 18:43:22.622681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.622642    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9196fb90-c915-443e-bd49-109a77351d03-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7\" (UID: \"9196fb90-c915-443e-bd49-109a77351d03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"
Oct 30 18:43:22.635504 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.635472    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4lv\" (UniqueName: \"kubernetes.io/projected/9196fb90-c915-443e-bd49-109a77351d03-kube-api-access-8c4lv\") pod \"payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7\" (UID: \"9196fb90-c915-443e-bd49-109a77351d03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"
Oct 30 18:43:22.708396 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.708348    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t" event={"ID":"23924268-e8d8-4506-8581-5c5ae5fc9ddf","Type":"ContainerStarted","Data":"4ca7ffd27e66e78dcd9c9b64a3e4871bd7d940c8e3cc1a55324ae431bc52eaa9"}
Oct 30 18:43:22.708895 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.708423    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t" event={"ID":"23924268-e8d8-4506-8581-5c5ae5fc9ddf","Type":"ContainerStarted","Data":"62a9eb50f75b1642d0e7c3d3ac22c8756c8e131756567dadbc60ad120b144457"}
Oct 30 18:43:22.708895 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.708502    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"
Oct 30 18:43:22.729455 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.729386    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t" podStartSLOduration=1.729365762 podStartE2EDuration="1.729365762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:22.728455777 +0000 UTC m=+616.596874107" watchObservedRunningTime="2026-10-30 18:43:22.729365762 +0000 UTC m=+616.597784092"
Oct 30 18:43:22.749471 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.749426    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"
Oct 30 18:43:23.007202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:22.986814    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:43:23.062975 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:23.062932    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"]
Oct 30 18:43:23.065483 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:43:23.065451    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9196fb90_c915_443e_bd49_109a77351d03.slice/crio-09215dbfa3c33478793ea87f2e4fc1eed359d22d1e2603f09b406ae02429585d WatchSource:0}: Error finding container 09215dbfa3c33478793ea87f2e4fc1eed359d22d1e2603f09b406ae02429585d: Status 404 returned error can't find the container with id 09215dbfa3c33478793ea87f2e4fc1eed359d22d1e2603f09b406ae02429585d
Oct 30 18:43:23.716804 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:23.716759    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7" event={"ID":"9196fb90-c915-443e-bd49-109a77351d03","Type":"ContainerStarted","Data":"7396b75ef46870db798a0db09cb742d2c715fe3d1df5dc51c920f00cd2fc0e8d"}
Oct 30 18:43:23.717336 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:23.716815    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7" event={"ID":"9196fb90-c915-443e-bd49-109a77351d03","Type":"ContainerStarted","Data":"09215dbfa3c33478793ea87f2e4fc1eed359d22d1e2603f09b406ae02429585d"}
Oct 30 18:43:23.740601 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:23.740525    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7" podStartSLOduration=1.740499594 podStartE2EDuration="1.740499594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:23.73853393 +0000 UTC m=+617.606952261" watchObservedRunningTime="2026-10-30 18:43:23.740499594 +0000 UTC m=+617.608917920"
Oct 30 18:43:24.739155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:24.738477    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"
Oct 30 18:43:24.845026 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:24.844977    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"]
Oct 30 18:43:24.849753 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:24.849715    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:24.859489 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:24.858798    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-84c32a-serving-cert\""
Oct 30 18:43:24.863631 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:24.863590    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"]
Oct 30 18:43:24.963997 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:24.956871    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7q4j\" (UniqueName: \"kubernetes.io/projected/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-kube-api-access-r7q4j\") pod \"maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:24.963997 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:24.963485    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-metrics-certs\") pod \"maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:24.963997 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:24.963641    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:24.963997 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:24.963781    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-maas-api-tls\") pod \"maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:25.065940 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:25.065829    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7q4j\" (UniqueName: \"kubernetes.io/projected/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-kube-api-access-r7q4j\") pod \"maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:25.065940 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:25.065893    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-metrics-certs\") pod \"maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:25.066209 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:25.065972    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:25.066209 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:25.066054    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-maas-api-tls\") pod \"maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:25.066283 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:43:25.066262    4372 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-serving-cert: secret "maas-api-e2e-worker-w0-84c32a-serving-cert" not found
Oct 30 18:43:25.066363 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:43:25.066345    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-maas-api-tls podName:f84cfd48-726a-4579-afcf-ddbe01ea2fdb nodeName:}" failed. No retries permitted until 2026-10-30 18:43:25.566320873 +0000 UTC m=+619.434739192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-maas-api-tls") pod "maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk" (UID: "f84cfd48-726a-4579-afcf-ddbe01ea2fdb") : secret "maas-api-e2e-worker-w0-84c32a-serving-cert" not found
Oct 30 18:43:25.067031 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:25.067002    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:25.072888 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:25.072850    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-metrics-certs\") pod \"maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:25.113172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:25.110551    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7q4j\" (UniqueName: \"kubernetes.io/projected/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-kube-api-access-r7q4j\") pod \"maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:25.572391 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:25.572324    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-maas-api-tls\") pod \"maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:25.575668 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:25.575625    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-maas-api-tls\") pod \"maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:25.768182 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:25.767757    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:26.175460 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:26.175402    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"]
Oct 30 18:43:26.176882 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:43:26.176689    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84cfd48_726a_4579_afcf_ddbe01ea2fdb.slice/crio-04deb78a82c7ff4cb0fc344940f30495ef7bae1d528fb45f0d7ad9bfe01cbd4d WatchSource:0}: Error finding container 04deb78a82c7ff4cb0fc344940f30495ef7bae1d528fb45f0d7ad9bfe01cbd4d: Status 404 returned error can't find the container with id 04deb78a82c7ff4cb0fc344940f30495ef7bae1d528fb45f0d7ad9bfe01cbd4d
Oct 30 18:43:26.753666 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:26.753621    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk" event={"ID":"f84cfd48-726a-4579-afcf-ddbe01ea2fdb","Type":"ContainerStarted","Data":"04deb78a82c7ff4cb0fc344940f30495ef7bae1d528fb45f0d7ad9bfe01cbd4d"}
Oct 30 18:43:27.779235 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:27.777522    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk" event={"ID":"f84cfd48-726a-4579-afcf-ddbe01ea2fdb","Type":"ContainerStarted","Data":"a789959828b3c01b927b17729844c79b14207dc1e0de070705e53703c4a5c1f9"}
Oct 30 18:43:27.779235 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:27.778782    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:27.834160 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:43:27.831436    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:43:27.854736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:27.852080    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk" podStartSLOduration=3.358738415 podStartE2EDuration="3.852046657s" totalImagesPullingTime="493.308242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:27.832947562 +0000 UTC m=+621.701365893" watchObservedRunningTime="2026-10-30 18:43:27.852046657 +0000 UTC m=+621.720464988"
Oct 30 18:43:28.572339 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:28.572296    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"
Oct 30 18:43:29.228164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:29.223275    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:43:30.511521 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:30.511434    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:43:30.512101 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:30.511547    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" podUID="72834257-61fb-4596-8b82-5917990f7469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:43:33.603420 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:33.603370    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:43:33.604113 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:33.603454    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw" podUID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:43:33.721725 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:33.721671    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"
Oct 30 18:43:33.861387 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:33.861241    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"]
Oct 30 18:43:33.869566 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:33.868873    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:33.879322 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:33.879273    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"]
Oct 30 18:43:33.933378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:33.933302    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-aa6736-serving-cert\""
Oct 30 18:43:33.997836 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:33.997387    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-metrics-certs\") pod \"maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:33.997836 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:33.997489    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8w4r\" (UniqueName: \"kubernetes.io/projected/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-kube-api-access-h8w4r\") pod \"maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:33.997836 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:33.997554    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-maas-api-tls\") pod \"maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:33.997836 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:33.997605    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:34.098468 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.098401    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-metrics-certs\") pod \"maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:34.098717 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.098487    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8w4r\" (UniqueName: \"kubernetes.io/projected/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-kube-api-access-h8w4r\") pod \"maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:34.098717 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.098549    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-maas-api-tls\") pod \"maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:34.098717 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.098599    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:34.098907 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:43:34.098711    4372 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-serving-cert: secret "maas-api-e2e-worker-w1-aa6736-serving-cert" not found
Oct 30 18:43:34.098907 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:43:34.098806    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-maas-api-tls podName:2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56 nodeName:}" failed. No retries permitted until 2026-10-30 18:43:34.598782555 +0000 UTC m=+628.467200877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-maas-api-tls") pod "maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn" (UID: "2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56") : secret "maas-api-e2e-worker-w1-aa6736-serving-cert" not found
Oct 30 18:43:34.099491 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.099459    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:34.101763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.101731    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-metrics-certs\") pod \"maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:34.112938 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.112837    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8w4r\" (UniqueName: \"kubernetes.io/projected/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-kube-api-access-h8w4r\") pod \"maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:34.607534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.606322    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-maas-api-tls\") pod \"maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:34.614982 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.614916    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-maas-api-tls\") pod \"maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:34.800943 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.800900    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:34.846840 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.846789    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 18:43:34.904054 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.903101    4372 generic.go:356] "Generic (PLEG): container finished" podID="72834257-61fb-4596-8b82-5917990f7469" containerID="af5ea31094e675d08611808658b96313d4e5fcd8dd305bd3c838b87a8ba746fc" exitCode=0
Oct 30 18:43:34.904054 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.903237    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" event={"ID":"72834257-61fb-4596-8b82-5917990f7469","Type":"ContainerDied","Data":"af5ea31094e675d08611808658b96313d4e5fcd8dd305bd3c838b87a8ba746fc"}
Oct 30 18:43:34.904054 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.903761    4372 scope.go:122] "RemoveContainer" containerID="af5ea31094e675d08611808658b96313d4e5fcd8dd305bd3c838b87a8ba746fc"
Oct 30 18:43:34.909448 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.908815    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:34.909448 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.908952    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:34.909448 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.909005    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:34.912606 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.912567    4372 generic.go:356] "Generic (PLEG): container finished" podID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerID="cc880ea57511314e5161ed64b11449474d54fe1e325a8c25bd0e81a3f734230c" exitCode=0
Oct 30 18:43:34.912753 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.912725    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw" event={"ID":"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b","Type":"ContainerDied","Data":"cc880ea57511314e5161ed64b11449474d54fe1e325a8c25bd0e81a3f734230c"}
Oct 30 18:43:34.913342 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.913308    4372 scope.go:122] "RemoveContainer" containerID="cc880ea57511314e5161ed64b11449474d54fe1e325a8c25bd0e81a3f734230c"
Oct 30 18:43:34.919172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.919061    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:34.919906 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.919850    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:34.920185 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:34.920159    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:35.115651 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:35.114255    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"]
Oct 30 18:43:35.122765 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:35.122406    4372 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:43:35.282282 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:35.282231    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:43:35.649372 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:35.649316    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:43:35.649794 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:35.649392    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:43:35.748385 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:35.748346    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"
Oct 30 18:43:35.952054 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:35.950393    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" event={"ID":"72834257-61fb-4596-8b82-5917990f7469","Type":"ContainerStarted","Data":"52c666532f661acef0f8c6ca8f2fb7755a6d2c9edcc8259f87de867be9bb2039"}
Oct 30 18:43:35.954882 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:35.954832    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw" event={"ID":"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b","Type":"ContainerStarted","Data":"ee2173f5e31d96f58981ecf5ec7938a27075928702d0fec5fd29404cfee18ea7"}
Oct 30 18:43:35.956897 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:35.956812    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:35.957536 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:35.957475    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:43:35.957995 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:35.957939    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" podUID="72834257-61fb-4596-8b82-5917990f7469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 18:43:35.964324 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:35.963862    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn" event={"ID":"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56","Type":"ContainerStarted","Data":"3fc087c0323837120364b8037fab21e88bad77aade2b9e95524b473844892fba"}
Oct 30 18:43:35.964324 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:35.963935    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn" event={"ID":"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56","Type":"ContainerStarted","Data":"f911e090f974d96bd9a3e26dd834c6e216516e48903a5722e1e05cea15614a4a"}
Oct 30 18:43:35.964324 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:35.964297    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:36.632622 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:36.632572    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:36.632830 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:36.632707    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:43:36.632830 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:36.632743    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw" podUID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 18:43:36.958021 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:36.957861    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:43:36.958021 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:36.957953    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" podUID="72834257-61fb-4596-8b82-5917990f7469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 18:43:37.632792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:37.632739    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:43:37.633054 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:37.632820    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw" podUID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 18:43:37.957667 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:37.957561    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:43:37.957667 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:37.957649    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" podUID="72834257-61fb-4596-8b82-5917990f7469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 18:43:38.632613 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:38.632559    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:43:38.633101 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:38.632637    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw" podUID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 18:43:38.957550 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:38.957410    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:43:38.957550 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:38.957504    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" podUID="72834257-61fb-4596-8b82-5917990f7469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 18:43:39.012159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:39.011295    4372 generic.go:356] "Generic (PLEG): container finished" podID="10111dcc-0436-4206-b798-d0bac119328a" containerID="704cc894c079ca154a3adda2c659990411c5e2ad0a197eccc1026637ff020c3e" exitCode=0
Oct 30 18:43:39.012159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:39.011548    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" event={"ID":"10111dcc-0436-4206-b798-d0bac119328a","Type":"ContainerDied","Data":"704cc894c079ca154a3adda2c659990411c5e2ad0a197eccc1026637ff020c3e"}
Oct 30 18:43:39.012497 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:39.012181    4372 scope.go:122] "RemoveContainer" containerID="704cc894c079ca154a3adda2c659990411c5e2ad0a197eccc1026637ff020c3e"
Oct 30 18:43:39.029938 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:39.015438    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:39.029938 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:39.015577    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:39.029938 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:39.015626    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:43:39.049215 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:39.049110    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn" podStartSLOduration=5.523957211 podStartE2EDuration="6.049086802s" totalImagesPullingTime="525.129591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:36.130579538 +0000 UTC m=+629.998997902" watchObservedRunningTime="2026-10-30 18:43:39.049086802 +0000 UTC m=+632.917505133"
Oct 30 18:43:39.633159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:39.632720    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:43:39.633159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:39.632803    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw" podUID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 18:43:39.961249 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:39.960494    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:43:39.961249 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:39.960604    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" podUID="72834257-61fb-4596-8b82-5917990f7469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 18:43:40.045218 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:40.045103    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" event={"ID":"10111dcc-0436-4206-b798-d0bac119328a","Type":"ContainerStarted","Data":"4d8a3e297e336a81c8c4866e17921957b58235f6d5a346893c41673517b79106"}
Oct 30 18:43:40.636159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:40.633722    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:43:40.636159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:40.633804    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw" podUID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 18:43:40.949254 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:40.948640    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:40.949254 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:40.948821    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:43:40.949254 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:40.948873    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 18:43:40.964157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:40.957337    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:43:40.964157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:40.957438    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" podUID="72834257-61fb-4596-8b82-5917990f7469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 18:43:41.632943 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:41.632896    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:43:41.633339 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:41.633285    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw" podUID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 18:43:41.950317 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:41.948497    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:43:41.950317 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:41.948568    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 18:43:41.977689 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:41.977642    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 18:43:42.603674 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:42.603629    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:42.606287 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:42.606238    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:42.610074 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:42.610038    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 18:43:42.642025 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:42.641953    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:42.642723 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:42.642694    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:42.643768 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:42.643744    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 18:43:42.948091 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:42.947953    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:43:42.948091 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:42.948037    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 18:43:43.948477 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:43.948432    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:43:43.949189 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:43.948505    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 18:43:44.948674 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:44.948618    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:43:44.949226 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:44.948691    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 18:43:45.017787 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:45.017736    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:43:45.948661 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:45.948602    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:43:45.948939 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:45.948687    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 18:43:46.953367 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:46.953329    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:46.953908 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:46.953648    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:46.954741 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:46.954717    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 18:43:54.978250 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:43:54.978207    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:43:57.700819 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:43:57.700772    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:44:07.760797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:07.760754    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:44:14.469335 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.469284    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"]
Oct 30 18:44:14.475311 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.475265    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.502053 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.502012    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8a92cc\"/\"openshift-service-ca.crt\""
Oct 30 18:44:14.502053 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.502039    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8a92cc\"/\"kube-root-ca.crt\""
Oct 30 18:44:14.508846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.508794    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8a92cc\"/\"premium-w2-8a92cc-kserve-self-signed-certs\""
Oct 30 18:44:14.508846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.508843    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8a92cc\"/\"default-dockercfg-ttpl9\""
Oct 30 18:44:14.521754 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.521701    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"]
Oct 30 18:44:14.567021 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.566980    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-model-cache\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.567485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.567442    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-tmp-dir\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.567793 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.567778    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-home\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.568074 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.568034    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c272\" (UniqueName: \"kubernetes.io/projected/22579f9b-ea36-4a4b-867b-87019a0b62c6-kube-api-access-4c272\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.568313 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.568295    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22579f9b-ea36-4a4b-867b-87019a0b62c6-tls-certs\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.568547 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.568505    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-dshm\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.669779 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.669736    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-model-cache\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.669779 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.669786    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-tmp-dir\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.670078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.669823    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-home\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.670078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.669868    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c272\" (UniqueName: \"kubernetes.io/projected/22579f9b-ea36-4a4b-867b-87019a0b62c6-kube-api-access-4c272\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.670078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.669910    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22579f9b-ea36-4a4b-867b-87019a0b62c6-tls-certs\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.670078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.670033    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-dshm\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.670339 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.670313    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-tmp-dir\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.670401 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.670331    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-home\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.670440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.670414    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-model-cache\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.672816 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.672779    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-dshm\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.673194 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.673170    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22579f9b-ea36-4a4b-867b-87019a0b62c6-tls-certs\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.693553 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.693515    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c272\" (UniqueName: \"kubernetes.io/projected/22579f9b-ea36-4a4b-867b-87019a0b62c6-kube-api-access-4c272\") pod \"premium-w2-8a92cc-kserve-6797c69b66-jwbgv\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") " pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:14.788885 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:14.788751    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:15.095919 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:44:15.095868    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22579f9b_ea36_4a4b_867b_87019a0b62c6.slice/crio-e8251de367602a33c87b7d05de8d07cf07a795fba9b116f03b0dda89f0ca0d04 WatchSource:0}: Error finding container e8251de367602a33c87b7d05de8d07cf07a795fba9b116f03b0dda89f0ca0d04: Status 404 returned error can't find the container with id e8251de367602a33c87b7d05de8d07cf07a795fba9b116f03b0dda89f0ca0d04
Oct 30 18:44:15.098701 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:15.098666    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"]
Oct 30 18:44:15.325169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:15.325083    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" event={"ID":"22579f9b-ea36-4a4b-867b-87019a0b62c6","Type":"ContainerStarted","Data":"26be516e7f8b230308d5de26b5d4f556923c00c5d104e5c8c5c6257937103d0a"}
Oct 30 18:44:15.325420 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:15.325190    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" event={"ID":"22579f9b-ea36-4a4b-867b-87019a0b62c6","Type":"ContainerStarted","Data":"e8251de367602a33c87b7d05de8d07cf07a795fba9b116f03b0dda89f0ca0d04"}
Oct 30 18:44:15.353683 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:15.353556    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podStartSLOduration=1.35353676 podStartE2EDuration="1.35353676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:15.351321172 +0000 UTC m=+669.219739517" watchObservedRunningTime="2026-10-30 18:44:15.35353676 +0000 UTC m=+669.221955090"
Oct 30 18:44:15.789550 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:15.789438    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:19.595896 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:19.595798    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:44:24.789505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:24.789445    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:24.803474 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:24.803436    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:25.398295 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:25.398252    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:44:27.701182 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:44:27.701110    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:44:35.070007 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:35.069944    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:44:54.602574 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.602519    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"]
Oct 30 18:44:54.606961 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.606923    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.614853 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.614804    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-aa6736-openshift-default-dockercfg-9lfh6\""
Oct 30 18:44:54.623718 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.623680    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"]
Oct 30 18:44:54.694862 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.694800    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.694862 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.694864    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.695223 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.694960    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.695223 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.695067    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm2ws\" (UniqueName: \"kubernetes.io/projected/93367dd2-e078-4c5f-9e84-e8e5832bb87f-kube-api-access-fm2ws\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.695223 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.695114    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.695338 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.695260    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.695338 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.695315    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.695447 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.695350    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.695499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.695469    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.796332 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.796280    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.796589 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.796383    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.796589 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.796431    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.796589 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.796463    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.796589 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.796497    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.796817 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.796592    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.796817 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.796620    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.796817 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.796648    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.796817 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.796678    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm2ws\" (UniqueName: \"kubernetes.io/projected/93367dd2-e078-4c5f-9e84-e8e5832bb87f-kube-api-access-fm2ws\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.797036 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.796941    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.797036 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.796985    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.797249 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.797224    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.797382 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.797360    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.798029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.797995    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.799547 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.799519    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.799849 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.799831    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.810991 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.810958    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.811222 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.811194    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm2ws\" (UniqueName: \"kubernetes.io/projected/93367dd2-e078-4c5f-9e84-e8e5832bb87f-kube-api-access-fm2ws\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:54.924105 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:54.923989    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:55.218390 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:55.218349    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"]
Oct 30 18:44:55.220533 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:44:55.220489    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93367dd2_e078_4c5f_9e84_e8e5832bb87f.slice/crio-a9c35e3f9aae6639a06637917b233f75cd1774b6280698267169670ba47a29cd WatchSource:0}: Error finding container a9c35e3f9aae6639a06637917b233f75cd1774b6280698267169670ba47a29cd: Status 404 returned error can't find the container with id a9c35e3f9aae6639a06637917b233f75cd1774b6280698267169670ba47a29cd
Oct 30 18:44:55.223371 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:55.223330    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:44:55.223450 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:55.223413    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:44:55.223450 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:55.223442    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:44:55.562480 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:55.562352    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq" event={"ID":"93367dd2-e078-4c5f-9e84-e8e5832bb87f","Type":"ContainerStarted","Data":"498e0226236c63e590deab53af930a68805a63adfe4d7057c31e91d4d5c4d9c9"}
Oct 30 18:44:55.562480 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:55.562435    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq" event={"ID":"93367dd2-e078-4c5f-9e84-e8e5832bb87f","Type":"ContainerStarted","Data":"a9c35e3f9aae6639a06637917b233f75cd1774b6280698267169670ba47a29cd"}
Oct 30 18:44:55.592422 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:55.592346    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq" podStartSLOduration=1.592324143 podStartE2EDuration="1.592324143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:55.590667019 +0000 UTC m=+709.459085357" watchObservedRunningTime="2026-10-30 18:44:55.592324143 +0000 UTC m=+709.460742473"
Oct 30 18:44:55.924560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:55.924510    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:44:56.307553 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:56.307434    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:44:56.932199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:56.932151    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 18:44:56.932720 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:56.932234    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq" podUID="93367dd2-e078-4c5f-9e84-e8e5832bb87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 18:44:57.734088 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:44:57.734040    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:44:57.925374 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:57.925327    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 18:44:57.925614 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:57.925393    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq" podUID="93367dd2-e078-4c5f-9e84-e8e5832bb87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 18:44:58.932384 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:58.925655    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 18:44:58.932384 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:58.925738    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq" podUID="93367dd2-e078-4c5f-9e84-e8e5832bb87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 18:44:59.924614 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:59.924570    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 18:44:59.924842 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:44:59.924643    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq" podUID="93367dd2-e078-4c5f-9e84-e8e5832bb87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 18:45:00.205542 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:00.205430    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"]
Oct 30 18:45:00.211352 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:00.211310    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"
Oct 30 18:45:00.273372 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:00.272561    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af7c6c6-4162-4b9a-922c-bbc6d0957f89-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889765-w5zqz\" (UID: \"5af7c6c6-4162-4b9a-922c-bbc6d0957f89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"
Oct 30 18:45:00.275066 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:00.275030    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8km9n\" (UniqueName: \"kubernetes.io/projected/5af7c6c6-4162-4b9a-922c-bbc6d0957f89-kube-api-access-8km9n\") pod \"maas-api-key-cleanup-29889765-w5zqz\" (UID: \"5af7c6c6-4162-4b9a-922c-bbc6d0957f89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"
Oct 30 18:45:00.300162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:00.278242    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"]
Oct 30 18:45:00.300592 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:00.298623    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:45:00.377726 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:00.377658    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8km9n\" (UniqueName: \"kubernetes.io/projected/5af7c6c6-4162-4b9a-922c-bbc6d0957f89-kube-api-access-8km9n\") pod \"maas-api-key-cleanup-29889765-w5zqz\" (UID: \"5af7c6c6-4162-4b9a-922c-bbc6d0957f89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"
Oct 30 18:45:00.377986 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:00.377810    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af7c6c6-4162-4b9a-922c-bbc6d0957f89-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889765-w5zqz\" (UID: \"5af7c6c6-4162-4b9a-922c-bbc6d0957f89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"
Oct 30 18:45:00.403216 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:00.378768    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af7c6c6-4162-4b9a-922c-bbc6d0957f89-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889765-w5zqz\" (UID: \"5af7c6c6-4162-4b9a-922c-bbc6d0957f89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"
Oct 30 18:45:00.407010 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:00.406968    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8km9n\" (UniqueName: \"kubernetes.io/projected/5af7c6c6-4162-4b9a-922c-bbc6d0957f89-kube-api-access-8km9n\") pod \"maas-api-key-cleanup-29889765-w5zqz\" (UID: \"5af7c6c6-4162-4b9a-922c-bbc6d0957f89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"
Oct 30 18:45:00.527723 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:00.527619    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"
Oct 30 18:45:00.863289 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:00.863198    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"]
Oct 30 18:45:00.865225 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:45:00.865190    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af7c6c6_4162_4b9a_922c_bbc6d0957f89.slice/crio-33bb06752258c03af0d940a030d72b9e3d1e0f3629e2136ebdfc954e96e104bb WatchSource:0}: Error finding container 33bb06752258c03af0d940a030d72b9e3d1e0f3629e2136ebdfc954e96e104bb: Status 404 returned error can't find the container with id 33bb06752258c03af0d940a030d72b9e3d1e0f3629e2136ebdfc954e96e104bb
Oct 30 18:45:00.924950 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:00.924905    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 18:45:00.925228 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:00.924987    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq" podUID="93367dd2-e078-4c5f-9e84-e8e5832bb87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 18:45:01.624582 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:01.624538    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz" event={"ID":"5af7c6c6-4162-4b9a-922c-bbc6d0957f89","Type":"ContainerStarted","Data":"33bb06752258c03af0d940a030d72b9e3d1e0f3629e2136ebdfc954e96e104bb"}
Oct 30 18:45:01.925418 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:01.925266    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 18:45:01.925418 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:01.925361    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq" podUID="93367dd2-e078-4c5f-9e84-e8e5832bb87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 18:45:02.156718 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.156662    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"]
Oct 30 18:45:02.161508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.161472    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.168804 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.168764    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-84c32a-openshift-default-dockercfg-k6dxc\""
Oct 30 18:45:02.178904 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.178796    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"]
Oct 30 18:45:02.274701 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.274637    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.274925 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.274764    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbdgh\" (UniqueName: \"kubernetes.io/projected/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-kube-api-access-kbdgh\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.274925 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.274807    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.274925 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.274835    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.274925 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.274891    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.275224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.274967    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.275224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.274984    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.275224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.275040    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.275224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.275067    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.375912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.375858    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.375912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.375914    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbdgh\" (UniqueName: \"kubernetes.io/projected/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-kube-api-access-kbdgh\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.376314 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.375945    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.376314 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.375964    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.376314 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.376023    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.376314 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.376097    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.376314 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.376162    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.376314 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.376235    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.376314 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.376268    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.376713 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.376436    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.376770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.376725    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.377319 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.377266    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.377477 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.377332    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.377477 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.377363    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.380619 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.380577    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.380827 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.380577    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.401326 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.401175    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbdgh\" (UniqueName: \"kubernetes.io/projected/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-kube-api-access-kbdgh\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.402556 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.402489    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.492290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.490463    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:02.849152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.849082    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"]
Oct 30 18:45:02.887495 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.877542    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:45:02.887495 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.877667    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:45:02.887495 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.877739    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:45:02.936299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.936260    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:45:02.941712 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.938714    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:45:02.941712 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:02.940225    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 18:45:03.645221 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:03.645061    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj" event={"ID":"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e","Type":"ContainerStarted","Data":"97703e073dc9a4c56d886cd133fb18219695dd6d5986b8ea2e88cae561410843"}
Oct 30 18:45:03.645221 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:03.645160    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj" event={"ID":"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e","Type":"ContainerStarted","Data":"116b638b4b78a7d1625d57a0a6ac1446e087336b5d4cbfdac2cf7af95d6fc9b1"}
Oct 30 18:45:03.716333 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:03.716250    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj" podStartSLOduration=1.716219761 podStartE2EDuration="1.716219761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.703382991 +0000 UTC m=+717.571801338" watchObservedRunningTime="2026-10-30 18:45:03.716219761 +0000 UTC m=+717.584638091"
Oct 30 18:45:04.503045 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:04.501671    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:04.505093 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:04.505032    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 18:45:04.505320 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:04.505147    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj" podUID="d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 18:45:05.495156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:05.494850    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 18:45:05.495156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:05.494983    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj" podUID="d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 18:45:05.656249 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:05.656178    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz" event={"ID":"5af7c6c6-4162-4b9a-922c-bbc6d0957f89","Type":"ContainerStarted","Data":"569b0ce2966bcc6092d51eefe65778e32edabfbdd67b916b7eda22e43616952c"}
Oct 30 18:45:05.705063 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:05.704965    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz" podStartSLOduration=1.905330589 podStartE2EDuration="5.704943876s" totalImagesPullingTime="3.799613287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:05.703485622 +0000 UTC m=+719.571903955" watchObservedRunningTime="2026-10-30 18:45:05.704943876 +0000 UTC m=+719.573362207"
Oct 30 18:45:06.503205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:06.501850    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 18:45:06.503205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:06.501943    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj" podUID="d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 18:45:06.663763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:06.663719    4372 generic.go:356] "Generic (PLEG): container finished" podID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89" containerID="569b0ce2966bcc6092d51eefe65778e32edabfbdd67b916b7eda22e43616952c" exitCode=60
Oct 30 18:45:06.664284 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:06.663801    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz" event={"ID":"5af7c6c6-4162-4b9a-922c-bbc6d0957f89","Type":"ContainerDied","Data":"569b0ce2966bcc6092d51eefe65778e32edabfbdd67b916b7eda22e43616952c"}
Oct 30 18:45:06.664284 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:06.664073    4372 scope.go:122] "RemoveContainer" containerID="569b0ce2966bcc6092d51eefe65778e32edabfbdd67b916b7eda22e43616952c"
Oct 30 18:45:07.498169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:07.496808    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 18:45:07.498169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:07.496887    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj" podUID="d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 18:45:07.672411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:07.672361    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz" event={"ID":"5af7c6c6-4162-4b9a-922c-bbc6d0957f89","Type":"ContainerStarted","Data":"212e644ad85934511fae83a4e338811a69968f0974d728dfdb3d334fd58fe7cd"}
Oct 30 18:45:08.495409 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.494713    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 18:45:08.495409 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.494840    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj" podUID="d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 18:45:08.692439 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.692395    4372 generic.go:356] "Generic (PLEG): container finished" podID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89" containerID="212e644ad85934511fae83a4e338811a69968f0974d728dfdb3d334fd58fe7cd" exitCode=60
Oct 30 18:45:08.699156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.693351    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz" event={"ID":"5af7c6c6-4162-4b9a-922c-bbc6d0957f89","Type":"ContainerDied","Data":"212e644ad85934511fae83a4e338811a69968f0974d728dfdb3d334fd58fe7cd"}
Oct 30 18:45:08.700520 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.696753    4372 scope.go:122] "RemoveContainer" containerID="212e644ad85934511fae83a4e338811a69968f0974d728dfdb3d334fd58fe7cd"
Oct 30 18:45:08.701278 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:45:08.701235    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889765-w5zqz_odh-ai-gateway-infra(5af7c6c6-4162-4b9a-922c-bbc6d0957f89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz" podUID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89"
Oct 30 18:45:08.701849 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.701820    4372 scope.go:122] "RemoveContainer" containerID="569b0ce2966bcc6092d51eefe65778e32edabfbdd67b916b7eda22e43616952c"
Oct 30 18:45:08.708975 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.708935    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"]
Oct 30 18:45:08.712830 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.712789    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"
Oct 30 18:45:08.720247 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.719847    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:45:08.728095 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.728005    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"]
Oct 30 18:45:08.807904 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.807756    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dbdac51-gmskc\" (UID: \"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"
Oct 30 18:45:08.807904 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.807880    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dbdac51-gmskc\" (UID: \"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"
Oct 30 18:45:08.808289 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.807994    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hggh6\" (UniqueName: \"kubernetes.io/projected/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-kube-api-access-hggh6\") pod \"maas-api-revoke-sub-7dbdac51-gmskc\" (UID: \"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"
Oct 30 18:45:08.928161 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.924099    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dbdac51-gmskc\" (UID: \"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"
Oct 30 18:45:08.928161 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.924289    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hggh6\" (UniqueName: \"kubernetes.io/projected/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-kube-api-access-hggh6\") pod \"maas-api-revoke-sub-7dbdac51-gmskc\" (UID: \"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"
Oct 30 18:45:08.928161 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.924379    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dbdac51-gmskc\" (UID: \"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"
Oct 30 18:45:08.928161 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.928092    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dbdac51-gmskc\" (UID: \"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"
Oct 30 18:45:08.934964 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.934892    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dbdac51-gmskc\" (UID: \"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"
Oct 30 18:45:08.956880 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:08.956785    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hggh6\" (UniqueName: \"kubernetes.io/projected/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-kube-api-access-hggh6\") pod \"maas-api-revoke-sub-7dbdac51-gmskc\" (UID: \"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"
Oct 30 18:45:09.039092 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:09.039034    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"
Oct 30 18:45:09.348963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:09.348926    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"]
Oct 30 18:45:09.351299 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:45:09.351251    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fa7e43_0cc5_4520_8d34_e20e611b5fbc.slice/crio-fb32166c57e841de46717366e3895223e37eb54aabbe4f8685e0955dfce4858c WatchSource:0}: Error finding container fb32166c57e841de46717366e3895223e37eb54aabbe4f8685e0955dfce4858c: Status 404 returned error can't find the container with id fb32166c57e841de46717366e3895223e37eb54aabbe4f8685e0955dfce4858c
Oct 30 18:45:09.494142 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:09.494089    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 18:45:09.494365 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:09.494170    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj" podUID="d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 18:45:09.701308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:09.701245    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc" event={"ID":"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc","Type":"ContainerStarted","Data":"fb32166c57e841de46717366e3895223e37eb54aabbe4f8685e0955dfce4858c"}
Oct 30 18:45:09.703596 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:09.703564    4372 scope.go:122] "RemoveContainer" containerID="212e644ad85934511fae83a4e338811a69968f0974d728dfdb3d334fd58fe7cd"
Oct 30 18:45:09.703912 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:45:09.703889    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889765-w5zqz_odh-ai-gateway-infra(5af7c6c6-4162-4b9a-922c-bbc6d0957f89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz" podUID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89"
Oct 30 18:45:10.499547 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:10.499499    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:10.499861 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:10.499840    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:10.500972 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:10.500941    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 18:45:10.711052 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:10.711013    4372 generic.go:356] "Generic (PLEG): container finished" podID="a9fa7e43-0cc5-4520-8d34-e20e611b5fbc" containerID="09cfb2931ade9b998deb956293d9b9e7b36ea5a3be9028566de3723f726031b7" exitCode=0
Oct 30 18:45:10.711786 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:10.711198    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc" event={"ID":"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc","Type":"ContainerDied","Data":"09cfb2931ade9b998deb956293d9b9e7b36ea5a3be9028566de3723f726031b7"}
Oct 30 18:45:11.919678 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:11.919635    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"
Oct 30 18:45:12.083052 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:12.083011    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-kube-api-access-hggh6\" (UniqueName: \"kubernetes.io/projected/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-kube-api-access-hggh6\") pod \"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc\" (UID: \"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc\") "
Oct 30 18:45:12.083312 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:12.083184    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-maas-api-xks-ca\") pod \"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc\" (UID: \"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc\") "
Oct 30 18:45:12.083312 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:12.083228    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-maas-api-openshift-ca\") pod \"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc\" (UID: \"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc\") "
Oct 30 18:45:12.083650 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:12.083622    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-maas-api-openshift-ca" pod "a9fa7e43-0cc5-4520-8d34-e20e611b5fbc" (UID: "a9fa7e43-0cc5-4520-8d34-e20e611b5fbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:45:12.083808 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:12.083787    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-maas-api-openshift-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:12.085615 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:12.085587    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-maas-api-xks-ca" pod "a9fa7e43-0cc5-4520-8d34-e20e611b5fbc" (UID: "a9fa7e43-0cc5-4520-8d34-e20e611b5fbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:45:12.085706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:12.085617    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-kube-api-access-hggh6" pod "a9fa7e43-0cc5-4520-8d34-e20e611b5fbc" (UID: "a9fa7e43-0cc5-4520-8d34-e20e611b5fbc"). InnerVolumeSpecName "kube-api-access-hggh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:12.185259 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:12.185217    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hggh6\" (UniqueName: \"kubernetes.io/projected/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-kube-api-access-hggh6\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:12.185259 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:12.185255    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc-maas-api-xks-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:12.727145 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:12.727102    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"
Oct 30 18:45:12.727353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:12.727108    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc" event={"ID":"a9fa7e43-0cc5-4520-8d34-e20e611b5fbc","Type":"ContainerDied","Data":"fb32166c57e841de46717366e3895223e37eb54aabbe4f8685e0955dfce4858c"}
Oct 30 18:45:12.727353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:12.727236    4372 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb32166c57e841de46717366e3895223e37eb54aabbe4f8685e0955dfce4858c"
Oct 30 18:45:13.939160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:13.938308    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"]
Oct 30 18:45:13.939755 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:13.939284    4372 cpu_manager.go:414] "containerMap: removing container" podUID="a9fa7e43-0cc5-4520-8d34-e20e611b5fbc" containerName="revoke-keys"
Oct 30 18:45:13.939755 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:13.939306    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fa7e43-0cc5-4520-8d34-e20e611b5fbc" containerName="revoke-keys"
Oct 30 18:45:13.939755 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:13.939432    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fa7e43-0cc5-4520-8d34-e20e611b5fbc" containerName="revoke-keys"
Oct 30 18:45:13.946112 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:13.943850    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"
Oct 30 18:45:13.950569 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:13.947716    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"]
Oct 30 18:45:13.954215 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:13.952348    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:45:14.111329 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:14.111259    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvzsr\" (UniqueName: \"kubernetes.io/projected/a1581874-75a0-40ca-a7b8-0dc3a854b300-kube-api-access-rvzsr\") pod \"maas-api-revoke-sub-02159ea4-tvqbp\" (UID: \"a1581874-75a0-40ca-a7b8-0dc3a854b300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"
Oct 30 18:45:14.111542 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:14.111423    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1581874-75a0-40ca-a7b8-0dc3a854b300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02159ea4-tvqbp\" (UID: \"a1581874-75a0-40ca-a7b8-0dc3a854b300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"
Oct 30 18:45:14.111742 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:14.111713    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1581874-75a0-40ca-a7b8-0dc3a854b300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02159ea4-tvqbp\" (UID: \"a1581874-75a0-40ca-a7b8-0dc3a854b300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"
Oct 30 18:45:14.212846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:14.212700    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1581874-75a0-40ca-a7b8-0dc3a854b300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02159ea4-tvqbp\" (UID: \"a1581874-75a0-40ca-a7b8-0dc3a854b300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"
Oct 30 18:45:14.212846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:14.212795    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvzsr\" (UniqueName: \"kubernetes.io/projected/a1581874-75a0-40ca-a7b8-0dc3a854b300-kube-api-access-rvzsr\") pod \"maas-api-revoke-sub-02159ea4-tvqbp\" (UID: \"a1581874-75a0-40ca-a7b8-0dc3a854b300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"
Oct 30 18:45:14.213202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:14.212869    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1581874-75a0-40ca-a7b8-0dc3a854b300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02159ea4-tvqbp\" (UID: \"a1581874-75a0-40ca-a7b8-0dc3a854b300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"
Oct 30 18:45:14.213793 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:14.213761    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1581874-75a0-40ca-a7b8-0dc3a854b300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02159ea4-tvqbp\" (UID: \"a1581874-75a0-40ca-a7b8-0dc3a854b300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"
Oct 30 18:45:14.215918 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:14.215888    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1581874-75a0-40ca-a7b8-0dc3a854b300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02159ea4-tvqbp\" (UID: \"a1581874-75a0-40ca-a7b8-0dc3a854b300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"
Oct 30 18:45:14.227231 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:14.227187    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvzsr\" (UniqueName: \"kubernetes.io/projected/a1581874-75a0-40ca-a7b8-0dc3a854b300-kube-api-access-rvzsr\") pod \"maas-api-revoke-sub-02159ea4-tvqbp\" (UID: \"a1581874-75a0-40ca-a7b8-0dc3a854b300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"
Oct 30 18:45:14.273443 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:14.273390    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"
Oct 30 18:45:14.656290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:14.656250    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"]
Oct 30 18:45:14.658203 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:45:14.658163    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1581874_75a0_40ca_a7b8_0dc3a854b300.slice/crio-eae5b6fa4675799b2e305dc4d29734f9b83bd3aa86920d2163bbd805888c89b1 WatchSource:0}: Error finding container eae5b6fa4675799b2e305dc4d29734f9b83bd3aa86920d2163bbd805888c89b1: Status 404 returned error can't find the container with id eae5b6fa4675799b2e305dc4d29734f9b83bd3aa86920d2163bbd805888c89b1
Oct 30 18:45:14.740300 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:14.740182    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp" event={"ID":"a1581874-75a0-40ca-a7b8-0dc3a854b300","Type":"ContainerStarted","Data":"eae5b6fa4675799b2e305dc4d29734f9b83bd3aa86920d2163bbd805888c89b1"}
Oct 30 18:45:15.748371 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:15.748324    4372 generic.go:356] "Generic (PLEG): container finished" podID="a1581874-75a0-40ca-a7b8-0dc3a854b300" containerID="3748c3e0d855464e39edb46394504470c835cda942d04d090639e538ec9fb1d7" exitCode=0
Oct 30 18:45:15.748852 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:15.748410    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp" event={"ID":"a1581874-75a0-40ca-a7b8-0dc3a854b300","Type":"ContainerDied","Data":"3748c3e0d855464e39edb46394504470c835cda942d04d090639e538ec9fb1d7"}
Oct 30 18:45:16.922685 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:16.922649    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"
Oct 30 18:45:16.946117 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:16.946066    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1581874-75a0-40ca-a7b8-0dc3a854b300-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1581874-75a0-40ca-a7b8-0dc3a854b300-maas-api-openshift-ca\") pod \"a1581874-75a0-40ca-a7b8-0dc3a854b300\" (UID: \"a1581874-75a0-40ca-a7b8-0dc3a854b300\") "
Oct 30 18:45:16.946361 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:16.946281    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1581874-75a0-40ca-a7b8-0dc3a854b300-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1581874-75a0-40ca-a7b8-0dc3a854b300-maas-api-xks-ca\") pod \"a1581874-75a0-40ca-a7b8-0dc3a854b300\" (UID: \"a1581874-75a0-40ca-a7b8-0dc3a854b300\") "
Oct 30 18:45:16.946361 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:16.946337    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1581874-75a0-40ca-a7b8-0dc3a854b300-kube-api-access-rvzsr\" (UniqueName: \"kubernetes.io/projected/a1581874-75a0-40ca-a7b8-0dc3a854b300-kube-api-access-rvzsr\") pod \"a1581874-75a0-40ca-a7b8-0dc3a854b300\" (UID: \"a1581874-75a0-40ca-a7b8-0dc3a854b300\") "
Oct 30 18:45:16.946560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:16.946526    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1581874-75a0-40ca-a7b8-0dc3a854b300-maas-api-openshift-ca" pod "a1581874-75a0-40ca-a7b8-0dc3a854b300" (UID: "a1581874-75a0-40ca-a7b8-0dc3a854b300"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:45:16.946866 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:16.946833    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1581874-75a0-40ca-a7b8-0dc3a854b300-maas-api-openshift-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:16.949259 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:16.949227    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1581874-75a0-40ca-a7b8-0dc3a854b300-kube-api-access-rvzsr" pod "a1581874-75a0-40ca-a7b8-0dc3a854b300" (UID: "a1581874-75a0-40ca-a7b8-0dc3a854b300"). InnerVolumeSpecName "kube-api-access-rvzsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:16.949436 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:16.949223    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1581874-75a0-40ca-a7b8-0dc3a854b300-maas-api-xks-ca" pod "a1581874-75a0-40ca-a7b8-0dc3a854b300" (UID: "a1581874-75a0-40ca-a7b8-0dc3a854b300"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:45:17.047768 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:17.047628    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1581874-75a0-40ca-a7b8-0dc3a854b300-maas-api-xks-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:17.047768 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:17.047680    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvzsr\" (UniqueName: \"kubernetes.io/projected/a1581874-75a0-40ca-a7b8-0dc3a854b300-kube-api-access-rvzsr\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:17.762387 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:17.762347    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"
Oct 30 18:45:17.762387 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:17.762362    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp" event={"ID":"a1581874-75a0-40ca-a7b8-0dc3a854b300","Type":"ContainerDied","Data":"eae5b6fa4675799b2e305dc4d29734f9b83bd3aa86920d2163bbd805888c89b1"}
Oct 30 18:45:17.762671 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:17.762422    4372 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eae5b6fa4675799b2e305dc4d29734f9b83bd3aa86920d2163bbd805888c89b1"
Oct 30 18:45:21.258569 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:21.258521    4372 scope.go:122] "RemoveContainer" containerID="212e644ad85934511fae83a4e338811a69968f0974d728dfdb3d334fd58fe7cd"
Oct 30 18:45:21.806249 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:21.806194    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz" event={"ID":"5af7c6c6-4162-4b9a-922c-bbc6d0957f89","Type":"ContainerStarted","Data":"2110d07ce3f01f531811aa7d70b49be8037114a9c7297522c4ae05a349eb9e1e"}
Oct 30 18:45:22.814679 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:22.814640    4372 generic.go:356] "Generic (PLEG): container finished" podID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89" containerID="2110d07ce3f01f531811aa7d70b49be8037114a9c7297522c4ae05a349eb9e1e" exitCode=60
Oct 30 18:45:22.815324 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:22.814713    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz" event={"ID":"5af7c6c6-4162-4b9a-922c-bbc6d0957f89","Type":"ContainerDied","Data":"2110d07ce3f01f531811aa7d70b49be8037114a9c7297522c4ae05a349eb9e1e"}
Oct 30 18:45:22.815324 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:22.814784    4372 scope.go:122] "RemoveContainer" containerID="212e644ad85934511fae83a4e338811a69968f0974d728dfdb3d334fd58fe7cd"
Oct 30 18:45:22.815324 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:22.815300    4372 scope.go:122] "RemoveContainer" containerID="2110d07ce3f01f531811aa7d70b49be8037114a9c7297522c4ae05a349eb9e1e"
Oct 30 18:45:22.815674 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:45:22.815646    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889765-w5zqz_odh-ai-gateway-infra(5af7c6c6-4162-4b9a-922c-bbc6d0957f89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz" podUID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89"
Oct 30 18:45:22.837333 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:22.837293    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"]
Oct 30 18:45:24.018861 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:24.018820    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"
Oct 30 18:45:24.071037 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:24.070987    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af7c6c6-4162-4b9a-922c-bbc6d0957f89-kube-api-access-8km9n\" (UniqueName: \"kubernetes.io/projected/5af7c6c6-4162-4b9a-922c-bbc6d0957f89-kube-api-access-8km9n\") pod \"5af7c6c6-4162-4b9a-922c-bbc6d0957f89\" (UID: \"5af7c6c6-4162-4b9a-922c-bbc6d0957f89\") "
Oct 30 18:45:24.071325 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:24.071088    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af7c6c6-4162-4b9a-922c-bbc6d0957f89-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af7c6c6-4162-4b9a-922c-bbc6d0957f89-maas-api-service-ca\") pod \"5af7c6c6-4162-4b9a-922c-bbc6d0957f89\" (UID: \"5af7c6c6-4162-4b9a-922c-bbc6d0957f89\") "
Oct 30 18:45:24.071852 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:24.071811    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af7c6c6-4162-4b9a-922c-bbc6d0957f89-maas-api-service-ca" pod "5af7c6c6-4162-4b9a-922c-bbc6d0957f89" (UID: "5af7c6c6-4162-4b9a-922c-bbc6d0957f89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:45:24.075083 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:24.074476    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af7c6c6-4162-4b9a-922c-bbc6d0957f89-kube-api-access-8km9n" pod "5af7c6c6-4162-4b9a-922c-bbc6d0957f89" (UID: "5af7c6c6-4162-4b9a-922c-bbc6d0957f89"). InnerVolumeSpecName "kube-api-access-8km9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:24.173454 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:24.173389    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8km9n\" (UniqueName: \"kubernetes.io/projected/5af7c6c6-4162-4b9a-922c-bbc6d0957f89-kube-api-access-8km9n\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:24.173454 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:24.173440    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af7c6c6-4162-4b9a-922c-bbc6d0957f89-maas-api-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:24.831377 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:24.831329    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz" event={"ID":"5af7c6c6-4162-4b9a-922c-bbc6d0957f89","Type":"ContainerDied","Data":"33bb06752258c03af0d940a030d72b9e3d1e0f3629e2136ebdfc954e96e104bb"}
Oct 30 18:45:24.831640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:24.831415    4372 scope.go:122] "RemoveContainer" containerID="2110d07ce3f01f531811aa7d70b49be8037114a9c7297522c4ae05a349eb9e1e"
Oct 30 18:45:24.831640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:24.831426    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"
Oct 30 18:45:24.895247 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:24.895189    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"]
Oct 30 18:45:24.898060 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:24.898007    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-w5zqz"]
Oct 30 18:45:26.265101 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:26.265041    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89" path="/var/lib/kubelet/pods/5af7c6c6-4162-4b9a-922c-bbc6d0957f89/volumes"
Oct 30 18:45:27.709994 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:45:27.709951    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:45:28.001078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.000962    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"]
Oct 30 18:45:28.001680 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.001648    4372 cpu_manager.go:414] "containerMap: removing container" podUID="a1581874-75a0-40ca-a7b8-0dc3a854b300" containerName="revoke-keys"
Oct 30 18:45:28.001680 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.001683    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1581874-75a0-40ca-a7b8-0dc3a854b300" containerName="revoke-keys"
Oct 30 18:45:28.001917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.001714    4372 cpu_manager.go:414] "containerMap: removing container" podUID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89" containerName="cleanup"
Oct 30 18:45:28.001917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.001721    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89" containerName="cleanup"
Oct 30 18:45:28.001917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.001738    4372 cpu_manager.go:414] "containerMap: removing container" podUID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89" containerName="cleanup"
Oct 30 18:45:28.001917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.001746    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89" containerName="cleanup"
Oct 30 18:45:28.001917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.001897    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1581874-75a0-40ca-a7b8-0dc3a854b300" containerName="revoke-keys"
Oct 30 18:45:28.001917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.001914    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89" containerName="cleanup"
Oct 30 18:45:28.002228 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.001925    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89" containerName="cleanup"
Oct 30 18:45:28.009552 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.009502    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:28.018301 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.018269    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ca7ae300b884\""
Oct 30 18:45:28.018456 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.018386    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9e3d91552b8a\""
Oct 30 18:45:28.018520 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.018494    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-hggpv\""
Oct 30 18:45:28.020105 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.020075    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 18:45:28.020346 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.020249    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 18:45:28.020346 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.020078    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 18:45:28.021520 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.021445    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 18:45:28.023958 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.023917    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"]
Oct 30 18:45:28.138290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.138236    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-provider-credentials\") pod \"payload-processing-external-model-59c9b654c7-j8tm7\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:28.140939 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.140782    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ghpc\" (UniqueName: \"kubernetes.io/projected/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-kube-api-access-8ghpc\") pod \"payload-processing-external-model-59c9b654c7-j8tm7\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:28.141548 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.141507    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-plugins-config-volume\") pod \"payload-processing-external-model-59c9b654c7-j8tm7\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:28.141758 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.141743    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-routing-overlay-volume\") pod \"payload-processing-external-model-59c9b654c7-j8tm7\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:28.243399 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.243337    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ghpc\" (UniqueName: \"kubernetes.io/projected/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-kube-api-access-8ghpc\") pod \"payload-processing-external-model-59c9b654c7-j8tm7\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:28.243703 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.243462    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-plugins-config-volume\") pod \"payload-processing-external-model-59c9b654c7-j8tm7\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:28.243703 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.243500    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-routing-overlay-volume\") pod \"payload-processing-external-model-59c9b654c7-j8tm7\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:28.243703 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.243664    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-provider-credentials\") pod \"payload-processing-external-model-59c9b654c7-j8tm7\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:28.244717 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.244665    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-routing-overlay-volume\") pod \"payload-processing-external-model-59c9b654c7-j8tm7\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:28.244871 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.244819    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-plugins-config-volume\") pod \"payload-processing-external-model-59c9b654c7-j8tm7\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:28.247441 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.247380    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-provider-credentials\") pod \"payload-processing-external-model-59c9b654c7-j8tm7\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:28.259057 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.258930    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ghpc\" (UniqueName: \"kubernetes.io/projected/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-kube-api-access-8ghpc\") pod \"payload-processing-external-model-59c9b654c7-j8tm7\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:28.328820 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.328481    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:28.642232 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.642179    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"]
Oct 30 18:45:28.870611 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.870561    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7" event={"ID":"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5","Type":"ContainerStarted","Data":"e9f69841f2bee878aeb62ef5ac57589b0297b1d119b0ec70544a219e04358581"}
Oct 30 18:45:28.871334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.870636    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7" event={"ID":"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5","Type":"ContainerStarted","Data":"a442e85c59a51f527369eaea06643e40b08bde9952d57c2d954a2dbac62986e2"}
Oct 30 18:45:28.871334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:28.870798    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:34.696972 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:34.696891    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7" podStartSLOduration=7.696860769 podStartE2EDuration="7.696860769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:28.901686612 +0000 UTC m=+742.770104943" watchObservedRunningTime="2026-10-30 18:45:34.696860769 +0000 UTC m=+748.565279098"
Oct 30 18:45:34.697945 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:34.697917    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:45:39.880222 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:39.880176    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:45:46.950169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:46.947166    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-998xh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:45:46.950169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:46.947284    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh" podUID="ccc2f38a-84e5-43c3-98d3-0f39ea658a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:45:47.165447 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:47.165353    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qstgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:45:47.165749 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:47.165706    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx" podUID="ce1f4775-99b1-4896-b58f-348c46e3b5f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:45:50.347998 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.346263    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"]
Oct 30 18:45:50.347998 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.347099    4372 cpu_manager.go:414] "containerMap: removing container" podUID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89" containerName="cleanup"
Oct 30 18:45:50.347998 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.347141    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89" containerName="cleanup"
Oct 30 18:45:50.347998 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.347299    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af7c6c6-4162-4b9a-922c-bbc6d0957f89" containerName="cleanup"
Oct 30 18:45:50.358545 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.354441    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:45:50.370089 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.370042    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"]
Oct 30 18:45:50.374271 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.370966    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0a3bc032a5f4\""
Oct 30 18:45:50.374271 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.371034    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d81f7a10928e\""
Oct 30 18:45:50.413029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.412974    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdrff\" (UniqueName: \"kubernetes.io/projected/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-kube-api-access-sdrff\") pod \"payload-processing-external-model-6c99bb4bf9-gzrrt\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") " pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:45:50.413281 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.413043    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-plugins-config-volume\") pod \"payload-processing-external-model-6c99bb4bf9-gzrrt\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") " pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:45:50.413281 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.413153    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-routing-overlay-volume\") pod \"payload-processing-external-model-6c99bb4bf9-gzrrt\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") " pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:45:50.413360 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.413314    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-provider-credentials\") pod \"payload-processing-external-model-6c99bb4bf9-gzrrt\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") " pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:45:50.514470 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.514417    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-provider-credentials\") pod \"payload-processing-external-model-6c99bb4bf9-gzrrt\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") " pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:45:50.514745 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.514525    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdrff\" (UniqueName: \"kubernetes.io/projected/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-kube-api-access-sdrff\") pod \"payload-processing-external-model-6c99bb4bf9-gzrrt\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") " pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:45:50.514745 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.514554    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-plugins-config-volume\") pod \"payload-processing-external-model-6c99bb4bf9-gzrrt\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") " pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:45:50.514745 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.514612    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-routing-overlay-volume\") pod \"payload-processing-external-model-6c99bb4bf9-gzrrt\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") " pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:45:50.515402 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.515368    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-routing-overlay-volume\") pod \"payload-processing-external-model-6c99bb4bf9-gzrrt\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") " pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:45:50.515541 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.515497    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-plugins-config-volume\") pod \"payload-processing-external-model-6c99bb4bf9-gzrrt\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") " pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:45:50.517976 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.517939    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-provider-credentials\") pod \"payload-processing-external-model-6c99bb4bf9-gzrrt\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") " pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:45:50.548808 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.548746    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdrff\" (UniqueName: \"kubernetes.io/projected/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-kube-api-access-sdrff\") pod \"payload-processing-external-model-6c99bb4bf9-gzrrt\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") " pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:45:50.683172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.683027    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:45:50.976858 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:50.976814    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"]
Oct 30 18:45:50.978955 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:45:50.978916    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbff5e45_33ed_43cd_9483_9fdb34f0cf1e.slice/crio-190e8704b143116a3ffbcd8d8a82c1d4db54decdfccc76f8306a6c19e55d1170 WatchSource:0}: Error finding container 190e8704b143116a3ffbcd8d8a82c1d4db54decdfccc76f8306a6c19e55d1170: Status 404 returned error can't find the container with id 190e8704b143116a3ffbcd8d8a82c1d4db54decdfccc76f8306a6c19e55d1170
Oct 30 18:45:51.076590 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:51.076539    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt" event={"ID":"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e","Type":"ContainerStarted","Data":"aa87e43b1a812d1aef19caa6f978c55fdd75288fc1ff78ae7ece720bd963681c"}
Oct 30 18:45:51.076801 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:51.076613    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt" event={"ID":"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e","Type":"ContainerStarted","Data":"190e8704b143116a3ffbcd8d8a82c1d4db54decdfccc76f8306a6c19e55d1170"}
Oct 30 18:45:51.076801 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:51.076743    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:45:51.124114 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:51.124034    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt" podStartSLOduration=1.124016687 podStartE2EDuration="1.124016687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:51.120853644 +0000 UTC m=+764.989271975" watchObservedRunningTime="2026-10-30 18:45:51.124016687 +0000 UTC m=+764.992435016"
Oct 30 18:45:51.834498 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:51.826740    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:45:55.465729 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.465680    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"]
Oct 30 18:45:55.469929 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.469897    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.480112 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.480078    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7a8fa2\"/\"openshift-service-ca.crt\""
Oct 30 18:45:55.480112 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.480099    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7a8fa2\"/\"kube-root-ca.crt\""
Oct 30 18:45:55.480350 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.480197    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7a8fa2\"/\"default-dockercfg-f2bkf\""
Oct 30 18:45:55.483534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.483508    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7a8fa2\"/\"e2e-conflict-a-89b77b0e-kserve-self-signed-certs\""
Oct 30 18:45:55.490266 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.490226    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"]
Oct 30 18:45:55.585396 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.585347    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e81165-3be3-4050-bbf1-0452b618fbb2-tls-certs\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.585612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.585410    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-model-cache\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.585612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.585467    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjdfg\" (UniqueName: \"kubernetes.io/projected/94e81165-3be3-4050-bbf1-0452b618fbb2-kube-api-access-fjdfg\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.585612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.585601    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-tmp-dir\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.585717 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.585654    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-home\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.585754 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.585741    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-dshm\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.687888 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.687835    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-model-cache\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.688212 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.687895    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjdfg\" (UniqueName: \"kubernetes.io/projected/94e81165-3be3-4050-bbf1-0452b618fbb2-kube-api-access-fjdfg\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.688212 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.687982    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-tmp-dir\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.688212 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.688062    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-home\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.688212 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.688160    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-dshm\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.688461 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.688260    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e81165-3be3-4050-bbf1-0452b618fbb2-tls-certs\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.689349 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.689314    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-tmp-dir\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.689630 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.689609    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-model-cache\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.693035 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.690696    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-home\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.694838 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.694785    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-dshm\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.696423 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.696377    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e81165-3be3-4050-bbf1-0452b618fbb2-tls-certs\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.704374 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.704339    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjdfg\" (UniqueName: \"kubernetes.io/projected/94e81165-3be3-4050-bbf1-0452b618fbb2-kube-api-access-fjdfg\") pod \"e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:55.786515 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:55.786380    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:56.088218 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.088169    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"]
Oct 30 18:45:56.092967 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:45:56.092912    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94e81165_3be3_4050_bbf1_0452b618fbb2.slice/crio-d23443d9ef043b392ed123dca0245ba7527e8aedb8365ca771985ecc201e6514 WatchSource:0}: Error finding container d23443d9ef043b392ed123dca0245ba7527e8aedb8365ca771985ecc201e6514: Status 404 returned error can't find the container with id d23443d9ef043b392ed123dca0245ba7527e8aedb8365ca771985ecc201e6514
Oct 30 18:45:56.093797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.093759    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"]
Oct 30 18:45:56.093973 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.093900    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.103492 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.103456    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-da70a5\"/\"kube-root-ca.crt\""
Oct 30 18:45:56.103708 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.103581    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-da70a5\"/\"openshift-service-ca.crt\""
Oct 30 18:45:56.103708 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.103643    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-da70a5\"/\"distinct-w4-da70a5-kserve-self-signed-certs\""
Oct 30 18:45:56.104371 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.104345    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-da70a5\"/\"default-dockercfg-v4s5d\""
Oct 30 18:45:56.106112 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.106077    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"]
Oct 30 18:45:56.121456 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.120003    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56" event={"ID":"94e81165-3be3-4050-bbf1-0452b618fbb2","Type":"ContainerStarted","Data":"d23443d9ef043b392ed123dca0245ba7527e8aedb8365ca771985ecc201e6514"}
Oct 30 18:45:56.194464 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.194415    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-home\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.194732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.194471    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-tmp-dir\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.194732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.194681    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h29md\" (UniqueName: \"kubernetes.io/projected/03a3b3a8-b37f-4c86-99fa-4e4494987be7-kube-api-access-h29md\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.194835 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.194739    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-dshm\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.194907 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.194834    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-model-cache\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.194907 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.194876    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03a3b3a8-b37f-4c86-99fa-4e4494987be7-tls-certs\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.301571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.299629    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-model-cache\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.301571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.299682    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03a3b3a8-b37f-4c86-99fa-4e4494987be7-tls-certs\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.301571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.299845    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-home\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.301571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.299876    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-tmp-dir\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.301571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.300011    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29md\" (UniqueName: \"kubernetes.io/projected/03a3b3a8-b37f-4c86-99fa-4e4494987be7-kube-api-access-h29md\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.301571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.300047    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-dshm\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.301571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.300754    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-model-cache\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.308277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.304784    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-home\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.308277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.305092    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-tmp-dir\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.317417 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.317012    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03a3b3a8-b37f-4c86-99fa-4e4494987be7-tls-certs\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.321503 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.320474    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-dshm\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.321768 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.321544    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h29md\" (UniqueName: \"kubernetes.io/projected/03a3b3a8-b37f-4c86-99fa-4e4494987be7-kube-api-access-h29md\") pod \"distinct-w4-da70a5-kserve-54f75954bc-hdgpb\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") " pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.424547 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.424432    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:56.713692 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:56.713652    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"]
Oct 30 18:45:56.715611 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:45:56.715557    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03a3b3a8_b37f_4c86_99fa_4e4494987be7.slice/crio-d0523b2b5b4abf05943571932fa79387f57880b02035e14740c8c02cea1ee8bb WatchSource:0}: Error finding container d0523b2b5b4abf05943571932fa79387f57880b02035e14740c8c02cea1ee8bb: Status 404 returned error can't find the container with id d0523b2b5b4abf05943571932fa79387f57880b02035e14740c8c02cea1ee8bb
Oct 30 18:45:57.127916 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:57.127850    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56" event={"ID":"94e81165-3be3-4050-bbf1-0452b618fbb2","Type":"ContainerStarted","Data":"a4f80ce9e3c3c570b3ec56365ec61d52522f826327897b852ccf119f4903bb9b"}
Oct 30 18:45:57.129715 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:57.129678    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" event={"ID":"03a3b3a8-b37f-4c86-99fa-4e4494987be7","Type":"ContainerStarted","Data":"46bc168ea2612f86f256fe3d7440fd6eb3acee8f7a167aa5ccfca5715c9a2b02"}
Oct 30 18:45:57.129966 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:57.129726    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" event={"ID":"03a3b3a8-b37f-4c86-99fa-4e4494987be7","Type":"ContainerStarted","Data":"d0523b2b5b4abf05943571932fa79387f57880b02035e14740c8c02cea1ee8bb"}
Oct 30 18:45:57.150337 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:57.150254    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56" podStartSLOduration=2.150208102 podStartE2EDuration="2.150208102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:57.148738399 +0000 UTC m=+771.017156728" watchObservedRunningTime="2026-10-30 18:45:57.150208102 +0000 UTC m=+771.018626432"
Oct 30 18:45:57.172335 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:57.172251    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podStartSLOduration=1.172227845 podStartE2EDuration="1.172227845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:57.169544149 +0000 UTC m=+771.037962481" watchObservedRunningTime="2026-10-30 18:45:57.172227845 +0000 UTC m=+771.040646211"
Oct 30 18:45:57.425480 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:57.425363    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:45:57.700543 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:45:57.700413    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:45:57.786874 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:57.786799    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:45:59.663157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.663094    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"]
Oct 30 18:45:59.667511 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.667475    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.700534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.700480    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"]
Oct 30 18:45:59.761646 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.761601    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.761646 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.761656    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a363b854-6fb7-4d52-805f-c6aff76b100a-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.761975 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.761792    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.761975 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.761830    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.761975 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.761939    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.762163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.762032    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqrmr\" (UniqueName: \"kubernetes.io/projected/a363b854-6fb7-4d52-805f-c6aff76b100a-kube-api-access-fqrmr\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.762163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.762138    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.762261 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.762219    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.762261 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.762252    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.863331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.863283    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.863331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.863333    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.863608 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.863361    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.863608 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.863415    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqrmr\" (UniqueName: \"kubernetes.io/projected/a363b854-6fb7-4d52-805f-c6aff76b100a-kube-api-access-fqrmr\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.863608 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.863441    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.863608 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.863488    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.863608 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.863516    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.863608 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.863595    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.863963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.863626    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a363b854-6fb7-4d52-805f-c6aff76b100a-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.863963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.863842    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.863963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.863951    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.864210 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.863964    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.864279 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.864259    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.864522 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.864502    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a363b854-6fb7-4d52-805f-c6aff76b100a-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.866551 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.866515    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.866714 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.866694    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.877961 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.877917    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.878188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.878013    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqrmr\" (UniqueName: \"kubernetes.io/projected/a363b854-6fb7-4d52-805f-c6aff76b100a-kube-api-access-fqrmr\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:45:59.983058 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:45:59.982925    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:46:00.308861 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:00.308820    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"]
Oct 30 18:46:00.312352 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:46:00.312316    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda363b854_6fb7_4d52_805f_c6aff76b100a.slice/crio-99656f31db899723457e2cc0cf9c956b0bbf767dca493a772e4ce8eae6747586 WatchSource:0}: Error finding container 99656f31db899723457e2cc0cf9c956b0bbf767dca493a772e4ce8eae6747586: Status 404 returned error can't find the container with id 99656f31db899723457e2cc0cf9c956b0bbf767dca493a772e4ce8eae6747586
Oct 30 18:46:00.315304 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:00.315257    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:46:00.315443 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:00.315340    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:46:00.315443 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:00.315376    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:46:01.159503 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:01.159449    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx" event={"ID":"a363b854-6fb7-4d52-805f-c6aff76b100a","Type":"ContainerStarted","Data":"de6604fc739c10a50629510c44e3746cb4e2b53f0690ffb08bf51770ec90681c"}
Oct 30 18:46:01.160061 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:01.159560    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx" event={"ID":"a363b854-6fb7-4d52-805f-c6aff76b100a","Type":"ContainerStarted","Data":"99656f31db899723457e2cc0cf9c956b0bbf767dca493a772e4ce8eae6747586"}
Oct 30 18:46:01.205939 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:01.205854    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx" podStartSLOduration=2.20582673 podStartE2EDuration="2.20582673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:01.196789013 +0000 UTC m=+775.065207371" watchObservedRunningTime="2026-10-30 18:46:01.20582673 +0000 UTC m=+775.074245062"
Oct 30 18:46:01.988155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:01.987369    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:46:01.988155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:01.987576    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 18:46:01.988155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:01.987626    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx" podUID="a363b854-6fb7-4d52-805f-c6aff76b100a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 18:46:02.104388 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:02.104346    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:46:02.214394 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:02.214344    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"]
Oct 30 18:46:02.987433 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:02.986794    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 18:46:02.987433 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:02.986900    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx" podUID="a363b854-6fb7-4d52-805f-c6aff76b100a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 18:46:03.985389 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:03.984744    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 18:46:03.985389 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:03.984824    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx" podUID="a363b854-6fb7-4d52-805f-c6aff76b100a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 18:46:04.989163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:04.988864    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 18:46:04.989163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:04.988946    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx" podUID="a363b854-6fb7-4d52-805f-c6aff76b100a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 18:46:05.861161 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:05.860655    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:46:05.998157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:05.988007    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 18:46:06.035332 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:06.035262    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx" podUID="a363b854-6fb7-4d52-805f-c6aff76b100a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 18:46:06.166231 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:06.165738    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:46:06.599756 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:06.599707    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:46:06.974112 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:06.974006    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:46:07.012615 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.012570    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 18:46:07.013452 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.013397    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx" podUID="a363b854-6fb7-4d52-805f-c6aff76b100a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 18:46:07.060104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.060009    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:46:07.218716 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.218638    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7" podUID="af5ba0f3-131c-4b84-9337-aa6f58ea6cb5" containerName="payload-processing" containerID="cri-o://e9f69841f2bee878aeb62ef5ac57589b0297b1d119b0ec70544a219e04358581" gracePeriod=25
Oct 30 18:46:07.459351 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.458673    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:46:07.721346 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.721277    4372 generic.go:356] "Generic (PLEG): container finished" podID="af5ba0f3-131c-4b84-9337-aa6f58ea6cb5" containerID="e9f69841f2bee878aeb62ef5ac57589b0297b1d119b0ec70544a219e04358581" exitCode=0
Oct 30 18:46:07.721758 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.721718    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7" event={"ID":"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5","Type":"ContainerDied","Data":"e9f69841f2bee878aeb62ef5ac57589b0297b1d119b0ec70544a219e04358581"}
Oct 30 18:46:07.752069 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.751995    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:46:07.795502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.795361    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-plugins-config-volume\") pod \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") "
Oct 30 18:46:07.795502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.795435    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-provider-credentials\") pod \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") "
Oct 30 18:46:07.795839 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.795719    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-routing-overlay-volume\") pod \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") "
Oct 30 18:46:07.795839 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.795782    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-kube-api-access-8ghpc\" (UniqueName: \"kubernetes.io/projected/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-kube-api-access-8ghpc\") pod \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\" (UID: \"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5\") "
Oct 30 18:46:07.796246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.796209    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-routing-overlay-volume" pod "af5ba0f3-131c-4b84-9337-aa6f58ea6cb5" (UID: "af5ba0f3-131c-4b84-9337-aa6f58ea6cb5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:46:07.796584 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.796555    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:07.796817 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.796771    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-plugins-config-volume" pod "af5ba0f3-131c-4b84-9337-aa6f58ea6cb5" (UID: "af5ba0f3-131c-4b84-9337-aa6f58ea6cb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:46:07.802177 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.800404    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-kube-api-access-8ghpc" pod "af5ba0f3-131c-4b84-9337-aa6f58ea6cb5" (UID: "af5ba0f3-131c-4b84-9337-aa6f58ea6cb5"). InnerVolumeSpecName "kube-api-access-8ghpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:07.802177 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.800471    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-provider-credentials" pod "af5ba0f3-131c-4b84-9337-aa6f58ea6cb5" (UID: "af5ba0f3-131c-4b84-9337-aa6f58ea6cb5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:07.898199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.898150    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ghpc\" (UniqueName: \"kubernetes.io/projected/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-kube-api-access-8ghpc\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:07.898199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.898195    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:07.898199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.898207    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:07.983923 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.983881    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 18:46:07.984165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:07.983951    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx" podUID="a363b854-6fb7-4d52-805f-c6aff76b100a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 18:46:08.731371 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:08.731335    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"
Oct 30 18:46:08.731971 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:08.731335    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7" event={"ID":"af5ba0f3-131c-4b84-9337-aa6f58ea6cb5","Type":"ContainerDied","Data":"a442e85c59a51f527369eaea06643e40b08bde9952d57c2d954a2dbac62986e2"}
Oct 30 18:46:08.731971 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:08.731476    4372 scope.go:122] "RemoveContainer" containerID="e9f69841f2bee878aeb62ef5ac57589b0297b1d119b0ec70544a219e04358581"
Oct 30 18:46:08.755221 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:08.755166    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"]
Oct 30 18:46:08.760195 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:08.760149    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59c9b654c7-j8tm7"]
Oct 30 18:46:09.430293 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:09.430255    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:46:09.430759 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:09.430730    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:46:09.432750 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:09.432703    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:46:10.266019 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:10.265981    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af5ba0f3-131c-4b84-9337-aa6f58ea6cb5" path="/var/lib/kubelet/pods/af5ba0f3-131c-4b84-9337-aa6f58ea6cb5/volumes"
Oct 30 18:46:12.427505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:12.427455    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:46:23.959757 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:23.959696    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"]
Oct 30 18:46:23.960544 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:23.960513    4372 cpu_manager.go:414] "containerMap: removing container" podUID="af5ba0f3-131c-4b84-9337-aa6f58ea6cb5" containerName="payload-processing"
Oct 30 18:46:23.960648 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:23.960550    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af5ba0f3-131c-4b84-9337-aa6f58ea6cb5" containerName="payload-processing"
Oct 30 18:46:23.960744 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:23.960728    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="af5ba0f3-131c-4b84-9337-aa6f58ea6cb5" containerName="payload-processing"
Oct 30 18:46:23.964565 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:23.964532    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:23.983108 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:23.983042    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-model-cache\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:23.983412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:23.983245    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhg7z\" (UniqueName: \"kubernetes.io/projected/2133ffc5-8d9a-4a88-a24f-96ba45031983-kube-api-access-mhg7z\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:23.983412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:23.983279    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-tmp-dir\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:23.983510 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:23.983404    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-home\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:23.983510 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:23.983460    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2133ffc5-8d9a-4a88-a24f-96ba45031983-tls-certs\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:23.983597 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:23.983536    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-dshm\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:23.987169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:23.987100    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7a8fa2\"/\"e2e-conflict-b-89b77b0e-kserve-self-signed-certs\""
Oct 30 18:46:23.991225 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:23.991175    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"]
Oct 30 18:46:24.085347 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.085292    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-model-cache\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:24.085586 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.085370    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhg7z\" (UniqueName: \"kubernetes.io/projected/2133ffc5-8d9a-4a88-a24f-96ba45031983-kube-api-access-mhg7z\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:24.085586 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.085403    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-tmp-dir\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:24.085586 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.085441    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-home\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:24.085586 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.085465    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2133ffc5-8d9a-4a88-a24f-96ba45031983-tls-certs\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:24.085586 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.085518    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-dshm\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:24.085894 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.085866    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-model-cache\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:24.085965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.085877    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-home\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:24.085965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.085943    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-tmp-dir\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:24.088547 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.088507    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-dshm\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:24.088815 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.088790    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2133ffc5-8d9a-4a88-a24f-96ba45031983-tls-certs\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:24.100115 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.100077    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhg7z\" (UniqueName: \"kubernetes.io/projected/2133ffc5-8d9a-4a88-a24f-96ba45031983-kube-api-access-mhg7z\") pod \"e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") " pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:24.280795 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.279983    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:24.624199 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:46:24.624150    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2133ffc5_8d9a_4a88_a24f_96ba45031983.slice/crio-b1e7e812550ab699bb935b835fda96809aeaa707366519709c4237604897fed9 WatchSource:0}: Error finding container b1e7e812550ab699bb935b835fda96809aeaa707366519709c4237604897fed9: Status 404 returned error can't find the container with id b1e7e812550ab699bb935b835fda96809aeaa707366519709c4237604897fed9
Oct 30 18:46:24.653175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.653093    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"]
Oct 30 18:46:24.839003 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.838948    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp" event={"ID":"2133ffc5-8d9a-4a88-a24f-96ba45031983","Type":"ContainerStarted","Data":"a52a5112e18dcbd38816f9865c7187678983391ba182493a67ef6faa936ef570"}
Oct 30 18:46:24.839290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.839018    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp" event={"ID":"2133ffc5-8d9a-4a88-a24f-96ba45031983","Type":"ContainerStarted","Data":"b1e7e812550ab699bb935b835fda96809aeaa707366519709c4237604897fed9"}
Oct 30 18:46:24.867544 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:24.867466    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp" podStartSLOduration=1.867442963 podStartE2EDuration="1.867442963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:24.865866746 +0000 UTC m=+798.734285077" watchObservedRunningTime="2026-10-30 18:46:24.867442963 +0000 UTC m=+798.735861293"
Oct 30 18:46:25.282683 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:25.282598    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:27.052166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.052100    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"]
Oct 30 18:46:27.056679 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.056646    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.071718 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.071672    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"]
Oct 30 18:46:27.144972 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.144908    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46273550-7eb9-47ba-8216-3d586b219ef3-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.144972 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.144980    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.145291 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.145073    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46273550-7eb9-47ba-8216-3d586b219ef3-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.145291 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.145220    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m5sj\" (UniqueName: \"kubernetes.io/projected/46273550-7eb9-47ba-8216-3d586b219ef3-kube-api-access-9m5sj\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.145291 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.145266    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.145291 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.145289    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.145427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.145347    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46273550-7eb9-47ba-8216-3d586b219ef3-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.145462 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.145448    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.145497 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.145477    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.246736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.246673    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46273550-7eb9-47ba-8216-3d586b219ef3-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.246736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.246742    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.247014 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.246889    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46273550-7eb9-47ba-8216-3d586b219ef3-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.247014 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.246982    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m5sj\" (UniqueName: \"kubernetes.io/projected/46273550-7eb9-47ba-8216-3d586b219ef3-kube-api-access-9m5sj\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.247180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.247034    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.247180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.247071    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.247180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.247108    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46273550-7eb9-47ba-8216-3d586b219ef3-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.247180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.247158    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.247416 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.247283    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.247416 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.247318    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.247521 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.247444    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.247691 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.247669    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.247792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.247767    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.248025 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.247820    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46273550-7eb9-47ba-8216-3d586b219ef3-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.249875 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.249842    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.250118 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.250095    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46273550-7eb9-47ba-8216-3d586b219ef3-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.260495 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.260452    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46273550-7eb9-47ba-8216-3d586b219ef3-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.260743 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.260706    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m5sj\" (UniqueName: \"kubernetes.io/projected/46273550-7eb9-47ba-8216-3d586b219ef3-kube-api-access-9m5sj\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.381393 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.381339    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:27.702283 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:46:27.702172    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:46:27.776707 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.775882    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"]
Oct 30 18:46:27.782227 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.781328    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:46:27.782227 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.781449    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:46:27.782227 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.781496    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:46:27.889332 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:27.888914    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9" event={"ID":"46273550-7eb9-47ba-8216-3d586b219ef3","Type":"ContainerStarted","Data":"632982e79f900b11e4febaaa59d39195a6eb4f147b12d7dbce64e619ef6ed082"}
Oct 30 18:46:28.910481 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:28.910426    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9" event={"ID":"46273550-7eb9-47ba-8216-3d586b219ef3","Type":"ContainerStarted","Data":"eda093a54590c6fe28ede98d7a26b376e130648381ad9bca5c82426a9af594e7"}
Oct 30 18:46:28.942289 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:28.942217    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9" podStartSLOduration=1.9421970119999998 podStartE2EDuration="1.942197012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:28.940113638 +0000 UTC m=+802.808532000" watchObservedRunningTime="2026-10-30 18:46:28.942197012 +0000 UTC m=+802.810615350"
Oct 30 18:46:29.388626 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:29.383658    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:29.402151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:29.396584    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:46:29.402151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:29.396690    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9" podUID="46273550-7eb9-47ba-8216-3d586b219ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 18:46:30.412377 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:30.412331    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:46:30.413361 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:30.413320    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9" podUID="46273550-7eb9-47ba-8216-3d586b219ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 18:46:30.742227 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:30.741301    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:46:31.385809 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:31.385753    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:46:31.386213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:31.386178    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9" podUID="46273550-7eb9-47ba-8216-3d586b219ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 18:46:32.388163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:32.382939    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:46:32.388163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:32.383062    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9" podUID="46273550-7eb9-47ba-8216-3d586b219ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 18:46:33.401166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:33.383737    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:46:33.401166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:33.383817    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9" podUID="46273550-7eb9-47ba-8216-3d586b219ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 18:46:34.301895 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:34.294730    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:34.316001 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:34.315967    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:34.401161 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:34.399773    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:46:34.401161 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:34.399899    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9" podUID="46273550-7eb9-47ba-8216-3d586b219ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 18:46:35.024987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:35.024946    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:46:35.618043 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:35.617999    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:35.618662 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:35.618483    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:35.619498 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:35.619478    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:46:44.761110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:44.761064    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"]
Oct 30 18:46:44.766955 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:44.766921    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:44.793159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:44.793053    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"]
Oct 30 18:46:45.034579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.034478    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kknmg\" (UniqueName: \"kubernetes.io/projected/de5bad8c-7124-450f-83aa-93b92384bd7a-kube-api-access-kknmg\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.034579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.034546    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.034776 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.034607    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5bad8c-7124-450f-83aa-93b92384bd7a-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.034776 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.034642    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.034776 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.034705    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.034776 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.034735    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.034922 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.034907    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.034958 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.034939    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.034992 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.034981    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.141196 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.141105    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5bad8c-7124-450f-83aa-93b92384bd7a-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.141196 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.141201    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.141502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.141270    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.141502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.141304    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.145164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.141676    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.145164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.141727    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.145164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.141775    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.145164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.141829    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kknmg\" (UniqueName: \"kubernetes.io/projected/de5bad8c-7124-450f-83aa-93b92384bd7a-kube-api-access-kknmg\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.145164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.141871    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.145164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.142746    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.145164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.143631    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5bad8c-7124-450f-83aa-93b92384bd7a-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.145164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.143874    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.145164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.144105    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.149792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.149757    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.164598 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.164558    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.195155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.190053    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.207462 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.207387    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kknmg\" (UniqueName: \"kubernetes.io/projected/de5bad8c-7124-450f-83aa-93b92384bd7a-kube-api-access-kknmg\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.232726 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.227757    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.247696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.247601    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:45.661247 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.661101    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"]
Oct 30 18:46:45.773423 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.771765    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:46:45.773423 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.771892    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:46:45.773423 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.771941    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:46:45.850143 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.850076    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"]
Oct 30 18:46:45.887258 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.887221    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"]
Oct 30 18:46:45.887659 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.887587    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:45.965403 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.964934    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7edb2248-2f7d-4569-bc95-3f148dffb171-routing-overlay-volume\") pod \"payload-processing-external-model-59c9b654c7-rhgzx\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:45.965403 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.964991    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ppn\" (UniqueName: \"kubernetes.io/projected/7edb2248-2f7d-4569-bc95-3f148dffb171-kube-api-access-b4ppn\") pod \"payload-processing-external-model-59c9b654c7-rhgzx\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:45.965403 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.965078    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7edb2248-2f7d-4569-bc95-3f148dffb171-plugins-config-volume\") pod \"payload-processing-external-model-59c9b654c7-rhgzx\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:45.965403 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:45.965299    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7edb2248-2f7d-4569-bc95-3f148dffb171-provider-credentials\") pod \"payload-processing-external-model-59c9b654c7-rhgzx\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:46.069168 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:46.067266    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7edb2248-2f7d-4569-bc95-3f148dffb171-provider-credentials\") pod \"payload-processing-external-model-59c9b654c7-rhgzx\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:46.069168 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:46.067407    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7edb2248-2f7d-4569-bc95-3f148dffb171-routing-overlay-volume\") pod \"payload-processing-external-model-59c9b654c7-rhgzx\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:46.069168 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:46.067445    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ppn\" (UniqueName: \"kubernetes.io/projected/7edb2248-2f7d-4569-bc95-3f148dffb171-kube-api-access-b4ppn\") pod \"payload-processing-external-model-59c9b654c7-rhgzx\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:46.069168 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:46.067572    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7edb2248-2f7d-4569-bc95-3f148dffb171-plugins-config-volume\") pod \"payload-processing-external-model-59c9b654c7-rhgzx\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:46.069168 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:46.069072    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7edb2248-2f7d-4569-bc95-3f148dffb171-plugins-config-volume\") pod \"payload-processing-external-model-59c9b654c7-rhgzx\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:46.073154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:46.069874    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7edb2248-2f7d-4569-bc95-3f148dffb171-routing-overlay-volume\") pod \"payload-processing-external-model-59c9b654c7-rhgzx\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:46.078157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:46.076806    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7edb2248-2f7d-4569-bc95-3f148dffb171-provider-credentials\") pod \"payload-processing-external-model-59c9b654c7-rhgzx\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:46.122325 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:46.122272    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ppn\" (UniqueName: \"kubernetes.io/projected/7edb2248-2f7d-4569-bc95-3f148dffb171-kube-api-access-b4ppn\") pod \"payload-processing-external-model-59c9b654c7-rhgzx\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") " pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:46.134379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:46.134254    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t" event={"ID":"de5bad8c-7124-450f-83aa-93b92384bd7a","Type":"ContainerStarted","Data":"61d19758fc7984cb53918841652d655f013401382968bc14b7f209d550747dd3"}
Oct 30 18:46:46.135783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:46.134636    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t" event={"ID":"de5bad8c-7124-450f-83aa-93b92384bd7a","Type":"ContainerStarted","Data":"9d0acde1720c8efd8ba27c955cfb85f1ff4b9324689e81ec86baa04e28f8df3c"}
Oct 30 18:46:46.235596 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:46.235496    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:46.258221 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:46.249233    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:46.850155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:46.844575    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t" podStartSLOduration=2.844551525 podStartE2EDuration="2.844551525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:46.234680169 +0000 UTC m=+820.103098503" watchObservedRunningTime="2026-10-30 18:46:46.844551525 +0000 UTC m=+820.712969855"
Oct 30 18:46:46.850155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:46.847418    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"]
Oct 30 18:46:46.855139 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:46:46.855037    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7edb2248_2f7d_4569_bc95_3f148dffb171.slice/crio-ed5a1f25a4018008d23cc08e7b64ae3592a9681f519fa42e54456ab31f3326d7 WatchSource:0}: Error finding container ed5a1f25a4018008d23cc08e7b64ae3592a9681f519fa42e54456ab31f3326d7: Status 404 returned error can't find the container with id ed5a1f25a4018008d23cc08e7b64ae3592a9681f519fa42e54456ab31f3326d7
Oct 30 18:46:47.141972 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:47.141876    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx" event={"ID":"7edb2248-2f7d-4569-bc95-3f148dffb171","Type":"ContainerStarted","Data":"817e59d125060d2c41b3051c840c63e8c0d15e821748402c7714e9cd8ccb12b4"}
Oct 30 18:46:47.141972 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:47.141949    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx" event={"ID":"7edb2248-2f7d-4569-bc95-3f148dffb171","Type":"ContainerStarted","Data":"ed5a1f25a4018008d23cc08e7b64ae3592a9681f519fa42e54456ab31f3326d7"}
Oct 30 18:46:47.257157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:47.256288    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:46:47.257157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:47.256362    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t" podUID="de5bad8c-7124-450f-83aa-93b92384bd7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 18:46:48.148495 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:48.148461    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:48.248716 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:48.248677    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:46:48.248972 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:48.248934    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t" podUID="de5bad8c-7124-450f-83aa-93b92384bd7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 18:46:49.248752 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:49.248713    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:46:49.249247 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:49.248775    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t" podUID="de5bad8c-7124-450f-83aa-93b92384bd7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 18:46:49.606037 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:49.605976    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx" podStartSLOduration=4.605960248 podStartE2EDuration="4.605960248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:47.175228272 +0000 UTC m=+821.043646608" watchObservedRunningTime="2026-10-30 18:46:49.605960248 +0000 UTC m=+823.474378577"
Oct 30 18:46:49.606244 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:49.606143    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"]
Oct 30 18:46:50.254697 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:50.254537    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:46:50.254697 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:50.254603    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t" podUID="de5bad8c-7124-450f-83aa-93b92384bd7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 18:46:51.248880 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:51.248834    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:46:51.249165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:51.248909    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t" podUID="de5bad8c-7124-450f-83aa-93b92384bd7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 18:46:52.248313 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:52.248271    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:46:52.248747 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:52.248339    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t" podUID="de5bad8c-7124-450f-83aa-93b92384bd7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 18:46:53.248492 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:53.248443    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:46:53.248925 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:53.248520    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t" podUID="de5bad8c-7124-450f-83aa-93b92384bd7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 18:46:54.253476 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:54.253445    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:54.253934 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:54.253913    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:54.254714 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:54.254692    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:46:55.221539 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:55.221504    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"]
Oct 30 18:46:56.277942 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:56.277910    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:46:57.962157 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:46:57.961734    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:46:58.516224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:58.516191    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"]
Oct 30 18:46:58.641152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:58.638489    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"]
Oct 30 18:46:58.712794 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:58.712413    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56" podUID="94e81165-3be3-4050-bbf1-0452b618fbb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:58.804209 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:58.801367    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:46:59.406154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:59.388282    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 18:46:59.451158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:59.447203    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:59.793267 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:59.790232    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56" podUID="94e81165-3be3-4050-bbf1-0452b618fbb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:46:59.816204 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:46:59.811076    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:47:00.051211 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:00.043235    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp" podUID="2133ffc5-8d9a-4a88-a24f-96ba45031983" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 18:47:00.392151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:00.388749    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:00.428165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:00.427542    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:47:00.507377 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:00.490762    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:47:00.832213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:00.830445    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56" podUID="94e81165-3be3-4050-bbf1-0452b618fbb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 18:47:01.045808 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:01.045740    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp" podUID="2133ffc5-8d9a-4a88-a24f-96ba45031983" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:01.046846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:01.046819    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:47:01.263879 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:01.263806    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:47:01.273146 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:01.269405    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:47:03.517529 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.517490    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt" podUID="fbff5e45-33ed-43cd-9483-9fdb34f0cf1e" containerName="payload-processing" containerID="cri-o://aa87e43b1a812d1aef19caa6f978c55fdd75288fc1ff78ae7ece720bd963681c" gracePeriod=25
Oct 30 18:47:03.639296 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.639255    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx" podUID="7edb2248-2f7d-4569-bc95-3f148dffb171" containerName="payload-processing" containerID="cri-o://817e59d125060d2c41b3051c840c63e8c0d15e821748402c7714e9cd8ccb12b4" gracePeriod=25
Oct 30 18:47:03.788694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.788632    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:47:03.829839 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.829806    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-routing-overlay-volume\") pod \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") "
Oct 30 18:47:03.829995 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.829885    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-plugins-config-volume\") pod \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") "
Oct 30 18:47:03.830061 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.830001    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-kube-api-access-sdrff\" (UniqueName: \"kubernetes.io/projected/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-kube-api-access-sdrff\") pod \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") "
Oct 30 18:47:03.830061 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.830041    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-provider-credentials\") pod \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\" (UID: \"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e\") "
Oct 30 18:47:03.830213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.830182    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-routing-overlay-volume" pod "fbff5e45-33ed-43cd-9483-9fdb34f0cf1e" (UID: "fbff5e45-33ed-43cd-9483-9fdb34f0cf1e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:47:03.830360 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.830332    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-plugins-config-volume" pod "fbff5e45-33ed-43cd-9483-9fdb34f0cf1e" (UID: "fbff5e45-33ed-43cd-9483-9fdb34f0cf1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:47:03.830525 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.830509    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:03.830597 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.830527    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:03.833176 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.833153    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-provider-credentials" pod "fbff5e45-33ed-43cd-9483-9fdb34f0cf1e" (UID: "fbff5e45-33ed-43cd-9483-9fdb34f0cf1e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:03.833262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.833229    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-kube-api-access-sdrff" pod "fbff5e45-33ed-43cd-9483-9fdb34f0cf1e" (UID: "fbff5e45-33ed-43cd-9483-9fdb34f0cf1e"). InnerVolumeSpecName "kube-api-access-sdrff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:03.928969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.928943    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:47:03.932166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.932144    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:03.932242 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:03.932169    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdrff\" (UniqueName: \"kubernetes.io/projected/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e-kube-api-access-sdrff\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:04.033489 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.033458    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7edb2248-2f7d-4569-bc95-3f148dffb171-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7edb2248-2f7d-4569-bc95-3f148dffb171-provider-credentials\") pod \"7edb2248-2f7d-4569-bc95-3f148dffb171\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") "
Oct 30 18:47:04.033658 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.033496    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7edb2248-2f7d-4569-bc95-3f148dffb171-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7edb2248-2f7d-4569-bc95-3f148dffb171-plugins-config-volume\") pod \"7edb2248-2f7d-4569-bc95-3f148dffb171\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") "
Oct 30 18:47:04.033743 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.033724    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7edb2248-2f7d-4569-bc95-3f148dffb171-kube-api-access-b4ppn\" (UniqueName: \"kubernetes.io/projected/7edb2248-2f7d-4569-bc95-3f148dffb171-kube-api-access-b4ppn\") pod \"7edb2248-2f7d-4569-bc95-3f148dffb171\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") "
Oct 30 18:47:04.033777 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.033763    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7edb2248-2f7d-4569-bc95-3f148dffb171-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7edb2248-2f7d-4569-bc95-3f148dffb171-routing-overlay-volume\") pod \"7edb2248-2f7d-4569-bc95-3f148dffb171\" (UID: \"7edb2248-2f7d-4569-bc95-3f148dffb171\") "
Oct 30 18:47:04.033978 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.033937    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7edb2248-2f7d-4569-bc95-3f148dffb171-plugins-config-volume" pod "7edb2248-2f7d-4569-bc95-3f148dffb171" (UID: "7edb2248-2f7d-4569-bc95-3f148dffb171"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:47:04.034194 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.034173    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7edb2248-2f7d-4569-bc95-3f148dffb171-routing-overlay-volume" pod "7edb2248-2f7d-4569-bc95-3f148dffb171" (UID: "7edb2248-2f7d-4569-bc95-3f148dffb171"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:47:04.034319 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.034304    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7edb2248-2f7d-4569-bc95-3f148dffb171-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:04.034361 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.034322    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7edb2248-2f7d-4569-bc95-3f148dffb171-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:04.035912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.035891    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7edb2248-2f7d-4569-bc95-3f148dffb171-kube-api-access-b4ppn" pod "7edb2248-2f7d-4569-bc95-3f148dffb171" (UID: "7edb2248-2f7d-4569-bc95-3f148dffb171"). InnerVolumeSpecName "kube-api-access-b4ppn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:04.035969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.035934    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7edb2248-2f7d-4569-bc95-3f148dffb171-provider-credentials" pod "7edb2248-2f7d-4569-bc95-3f148dffb171" (UID: "7edb2248-2f7d-4569-bc95-3f148dffb171"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:04.135527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.135496    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4ppn\" (UniqueName: \"kubernetes.io/projected/7edb2248-2f7d-4569-bc95-3f148dffb171-kube-api-access-b4ppn\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:04.135527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.135525    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7edb2248-2f7d-4569-bc95-3f148dffb171-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:04.453082 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.453003    4372 generic.go:356] "Generic (PLEG): container finished" podID="fbff5e45-33ed-43cd-9483-9fdb34f0cf1e" containerID="aa87e43b1a812d1aef19caa6f978c55fdd75288fc1ff78ae7ece720bd963681c" exitCode=0
Oct 30 18:47:04.453082 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.453051    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"
Oct 30 18:47:04.453299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.453081    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt" event={"ID":"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e","Type":"ContainerDied","Data":"aa87e43b1a812d1aef19caa6f978c55fdd75288fc1ff78ae7ece720bd963681c"}
Oct 30 18:47:04.453299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.453143    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt" event={"ID":"fbff5e45-33ed-43cd-9483-9fdb34f0cf1e","Type":"ContainerDied","Data":"190e8704b143116a3ffbcd8d8a82c1d4db54decdfccc76f8306a6c19e55d1170"}
Oct 30 18:47:04.453299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.453175    4372 scope.go:122] "RemoveContainer" containerID="aa87e43b1a812d1aef19caa6f978c55fdd75288fc1ff78ae7ece720bd963681c"
Oct 30 18:47:04.454990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.454892    4372 generic.go:356] "Generic (PLEG): container finished" podID="7edb2248-2f7d-4569-bc95-3f148dffb171" containerID="817e59d125060d2c41b3051c840c63e8c0d15e821748402c7714e9cd8ccb12b4" exitCode=0
Oct 30 18:47:04.454990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.454958    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"
Oct 30 18:47:04.455228 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.454968    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx" event={"ID":"7edb2248-2f7d-4569-bc95-3f148dffb171","Type":"ContainerDied","Data":"817e59d125060d2c41b3051c840c63e8c0d15e821748402c7714e9cd8ccb12b4"}
Oct 30 18:47:04.455228 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.455199    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx" event={"ID":"7edb2248-2f7d-4569-bc95-3f148dffb171","Type":"ContainerDied","Data":"ed5a1f25a4018008d23cc08e7b64ae3592a9681f519fa42e54456ab31f3326d7"}
Oct 30 18:47:04.462408 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.462387    4372 scope.go:122] "RemoveContainer" containerID="aa87e43b1a812d1aef19caa6f978c55fdd75288fc1ff78ae7ece720bd963681c"
Oct 30 18:47:04.462676 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:47:04.462658    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa87e43b1a812d1aef19caa6f978c55fdd75288fc1ff78ae7ece720bd963681c\": container with ID starting with aa87e43b1a812d1aef19caa6f978c55fdd75288fc1ff78ae7ece720bd963681c not found: ID does not exist" containerID="aa87e43b1a812d1aef19caa6f978c55fdd75288fc1ff78ae7ece720bd963681c"
Oct 30 18:47:04.462727 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.462684    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa87e43b1a812d1aef19caa6f978c55fdd75288fc1ff78ae7ece720bd963681c"} err="failed to get container status \"aa87e43b1a812d1aef19caa6f978c55fdd75288fc1ff78ae7ece720bd963681c\": rpc error: code = NotFound desc = could not find container \"aa87e43b1a812d1aef19caa6f978c55fdd75288fc1ff78ae7ece720bd963681c\": container with ID starting with aa87e43b1a812d1aef19caa6f978c55fdd75288fc1ff78ae7ece720bd963681c not found: ID does not exist"
Oct 30 18:47:04.462727 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.462698    4372 scope.go:122] "RemoveContainer" containerID="817e59d125060d2c41b3051c840c63e8c0d15e821748402c7714e9cd8ccb12b4"
Oct 30 18:47:04.470633 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.470617    4372 scope.go:122] "RemoveContainer" containerID="817e59d125060d2c41b3051c840c63e8c0d15e821748402c7714e9cd8ccb12b4"
Oct 30 18:47:04.470961 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:47:04.470940    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817e59d125060d2c41b3051c840c63e8c0d15e821748402c7714e9cd8ccb12b4\": container with ID starting with 817e59d125060d2c41b3051c840c63e8c0d15e821748402c7714e9cd8ccb12b4 not found: ID does not exist" containerID="817e59d125060d2c41b3051c840c63e8c0d15e821748402c7714e9cd8ccb12b4"
Oct 30 18:47:04.471056 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.470966    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817e59d125060d2c41b3051c840c63e8c0d15e821748402c7714e9cd8ccb12b4"} err="failed to get container status \"817e59d125060d2c41b3051c840c63e8c0d15e821748402c7714e9cd8ccb12b4\": rpc error: code = NotFound desc = could not find container \"817e59d125060d2c41b3051c840c63e8c0d15e821748402c7714e9cd8ccb12b4\": container with ID starting with 817e59d125060d2c41b3051c840c63e8c0d15e821748402c7714e9cd8ccb12b4 not found: ID does not exist"
Oct 30 18:47:04.474264 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.474245    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"]
Oct 30 18:47:04.478795 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.478775    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59c9b654c7-rhgzx"]
Oct 30 18:47:04.491290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.491271    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"]
Oct 30 18:47:04.498390 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.498371    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c99bb4bf9-gzrrt"]
Oct 30 18:47:04.627237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.627207    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp" podUID="2133ffc5-8d9a-4a88-a24f-96ba45031983" containerName="main" containerID="cri-o://a52a5112e18dcbd38816f9865c7187678983391ba182493a67ef6faa936ef570" gracePeriod=45
Oct 30 18:47:04.923578 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:04.923549    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:47:05.052316 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.052237    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-home\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-home\") pod \"2133ffc5-8d9a-4a88-a24f-96ba45031983\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") "
Oct 30 18:47:05.052316 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.052268    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-dshm\") pod \"2133ffc5-8d9a-4a88-a24f-96ba45031983\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") "
Oct 30 18:47:05.052519 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.052345    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-tmp-dir\") pod \"2133ffc5-8d9a-4a88-a24f-96ba45031983\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") "
Oct 30 18:47:05.052519 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.052410    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2133ffc5-8d9a-4a88-a24f-96ba45031983-tls-certs\" (UniqueName: \"kubernetes.io/secret/2133ffc5-8d9a-4a88-a24f-96ba45031983-tls-certs\") pod \"2133ffc5-8d9a-4a88-a24f-96ba45031983\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") "
Oct 30 18:47:05.052519 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.052461    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-model-cache\") pod \"2133ffc5-8d9a-4a88-a24f-96ba45031983\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") "
Oct 30 18:47:05.052519 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.052501    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2133ffc5-8d9a-4a88-a24f-96ba45031983-kube-api-access-mhg7z\" (UniqueName: \"kubernetes.io/projected/2133ffc5-8d9a-4a88-a24f-96ba45031983-kube-api-access-mhg7z\") pod \"2133ffc5-8d9a-4a88-a24f-96ba45031983\" (UID: \"2133ffc5-8d9a-4a88-a24f-96ba45031983\") "
Oct 30 18:47:05.052720 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.052537    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-home" pod "2133ffc5-8d9a-4a88-a24f-96ba45031983" (UID: "2133ffc5-8d9a-4a88-a24f-96ba45031983"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:47:05.052720 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.052626    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-tmp-dir" pod "2133ffc5-8d9a-4a88-a24f-96ba45031983" (UID: "2133ffc5-8d9a-4a88-a24f-96ba45031983"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:47:05.052720 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.052696    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-model-cache" pod "2133ffc5-8d9a-4a88-a24f-96ba45031983" (UID: "2133ffc5-8d9a-4a88-a24f-96ba45031983"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:47:05.052988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.052970    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:05.053041 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.052996    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:05.053041 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.053013    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:05.054792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.054772    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-dshm" pod "2133ffc5-8d9a-4a88-a24f-96ba45031983" (UID: "2133ffc5-8d9a-4a88-a24f-96ba45031983"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:47:05.054906 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.054799    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2133ffc5-8d9a-4a88-a24f-96ba45031983-kube-api-access-mhg7z" pod "2133ffc5-8d9a-4a88-a24f-96ba45031983" (UID: "2133ffc5-8d9a-4a88-a24f-96ba45031983"). InnerVolumeSpecName "kube-api-access-mhg7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:05.054906 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.054841    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2133ffc5-8d9a-4a88-a24f-96ba45031983-tls-certs" pod "2133ffc5-8d9a-4a88-a24f-96ba45031983" (UID: "2133ffc5-8d9a-4a88-a24f-96ba45031983"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:47:05.154392 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.154314    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2133ffc5-8d9a-4a88-a24f-96ba45031983-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:05.154392 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.154335    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2133ffc5-8d9a-4a88-a24f-96ba45031983-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:05.154392 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.154347    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhg7z\" (UniqueName: \"kubernetes.io/projected/2133ffc5-8d9a-4a88-a24f-96ba45031983-kube-api-access-mhg7z\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:05.463394 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.463357    4372 generic.go:356] "Generic (PLEG): container finished" podID="2133ffc5-8d9a-4a88-a24f-96ba45031983" containerID="a52a5112e18dcbd38816f9865c7187678983391ba182493a67ef6faa936ef570" exitCode=0
Oct 30 18:47:05.463556 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.463405    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"
Oct 30 18:47:05.463556 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.463407    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp" event={"ID":"2133ffc5-8d9a-4a88-a24f-96ba45031983","Type":"ContainerDied","Data":"a52a5112e18dcbd38816f9865c7187678983391ba182493a67ef6faa936ef570"}
Oct 30 18:47:05.463556 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.463444    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp" event={"ID":"2133ffc5-8d9a-4a88-a24f-96ba45031983","Type":"ContainerDied","Data":"b1e7e812550ab699bb935b835fda96809aeaa707366519709c4237604897fed9"}
Oct 30 18:47:05.463556 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.463467    4372 scope.go:122] "RemoveContainer" containerID="a52a5112e18dcbd38816f9865c7187678983391ba182493a67ef6faa936ef570"
Oct 30 18:47:05.474689 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.474669    4372 scope.go:122] "RemoveContainer" containerID="a52a5112e18dcbd38816f9865c7187678983391ba182493a67ef6faa936ef570"
Oct 30 18:47:05.475202 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:47:05.475039    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52a5112e18dcbd38816f9865c7187678983391ba182493a67ef6faa936ef570\": container with ID starting with a52a5112e18dcbd38816f9865c7187678983391ba182493a67ef6faa936ef570 not found: ID does not exist" containerID="a52a5112e18dcbd38816f9865c7187678983391ba182493a67ef6faa936ef570"
Oct 30 18:47:05.475202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.475075    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52a5112e18dcbd38816f9865c7187678983391ba182493a67ef6faa936ef570"} err="failed to get container status \"a52a5112e18dcbd38816f9865c7187678983391ba182493a67ef6faa936ef570\": rpc error: code = NotFound desc = could not find container \"a52a5112e18dcbd38816f9865c7187678983391ba182493a67ef6faa936ef570\": container with ID starting with a52a5112e18dcbd38816f9865c7187678983391ba182493a67ef6faa936ef570 not found: ID does not exist"
Oct 30 18:47:05.486880 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.486858    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"]
Oct 30 18:47:05.492037 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:05.492018    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-b-89b77b0e-kserve-559f7b4bf5-jfpvp"]
Oct 30 18:47:06.266090 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.266055    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2133ffc5-8d9a-4a88-a24f-96ba45031983" path="/var/lib/kubelet/pods/2133ffc5-8d9a-4a88-a24f-96ba45031983/volumes"
Oct 30 18:47:06.266489 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.266472    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7edb2248-2f7d-4569-bc95-3f148dffb171" path="/var/lib/kubelet/pods/7edb2248-2f7d-4569-bc95-3f148dffb171/volumes"
Oct 30 18:47:06.266817 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.266801    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbff5e45-33ed-43cd-9483-9fdb34f0cf1e" path="/var/lib/kubelet/pods/fbff5e45-33ed-43cd-9483-9fdb34f0cf1e/volumes"
Oct 30 18:47:06.769559 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.769529    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"]
Oct 30 18:47:06.770218 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.770201    4372 cpu_manager.go:414] "containerMap: removing container" podUID="7edb2248-2f7d-4569-bc95-3f148dffb171" containerName="payload-processing"
Oct 30 18:47:06.770288 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.770222    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7edb2248-2f7d-4569-bc95-3f148dffb171" containerName="payload-processing"
Oct 30 18:47:06.770288 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.770263    4372 cpu_manager.go:414] "containerMap: removing container" podUID="fbff5e45-33ed-43cd-9483-9fdb34f0cf1e" containerName="payload-processing"
Oct 30 18:47:06.770288 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.770271    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbff5e45-33ed-43cd-9483-9fdb34f0cf1e" containerName="payload-processing"
Oct 30 18:47:06.770288 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.770281    4372 cpu_manager.go:414] "containerMap: removing container" podUID="2133ffc5-8d9a-4a88-a24f-96ba45031983" containerName="main"
Oct 30 18:47:06.770288 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.770288    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2133ffc5-8d9a-4a88-a24f-96ba45031983" containerName="main"
Oct 30 18:47:06.770471 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.770404    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="7edb2248-2f7d-4569-bc95-3f148dffb171" containerName="payload-processing"
Oct 30 18:47:06.770471 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.770421    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbff5e45-33ed-43cd-9483-9fdb34f0cf1e" containerName="payload-processing"
Oct 30 18:47:06.770471 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.770431    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="2133ffc5-8d9a-4a88-a24f-96ba45031983" containerName="main"
Oct 30 18:47:06.775482 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.775460    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:06.794804 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.794785    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7a8fa2\"/\"distinct-w3-7a8fa2-kserve-self-signed-certs\""
Oct 30 18:47:06.799082 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.799060    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"]
Oct 30 18:47:06.877424 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.877403    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-dshm\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:06.877511 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.877431    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-tmp-dir\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:06.877587 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.877569    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-home\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:06.877621 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.877597    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-tls-certs\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:06.877728 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.877717    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-model-cache\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:06.877763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.877736    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rf5b\" (UniqueName: \"kubernetes.io/projected/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-kube-api-access-6rf5b\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:06.983152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.980708    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-model-cache\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:06.983152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.980753    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rf5b\" (UniqueName: \"kubernetes.io/projected/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-kube-api-access-6rf5b\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:06.983152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.980832    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-dshm\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:06.983152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.980857    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-tmp-dir\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:06.983152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.980948    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-home\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:06.983152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.980974    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-tls-certs\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:06.983152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.982171    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-tmp-dir\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:06.983152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.982389    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-model-cache\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:06.983152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:06.982628    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-home\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:07.009526 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.009327    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-tls-certs\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:07.011829 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.011764    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-dshm\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:07.013272 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.013204    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rf5b\" (UniqueName: \"kubernetes.io/projected/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-kube-api-access-6rf5b\") pod \"distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:07.088529 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.088501    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:07.370099 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.370071    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"]
Oct 30 18:47:07.371006 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:47:07.370980    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod665d8ad1_52bf_4837_9a43_c21c4b2ea5ab.slice/crio-a2b72d11ea497ce692013440b6af20ea723ea2129733f432d3298af810f11828 WatchSource:0}: Error finding container a2b72d11ea497ce692013440b6af20ea723ea2129733f432d3298af810f11828: Status 404 returned error can't find the container with id a2b72d11ea497ce692013440b6af20ea723ea2129733f432d3298af810f11828
Oct 30 18:47:07.477707 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.477672    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" event={"ID":"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab","Type":"ContainerStarted","Data":"302e2cd731be5a952c9608682fded282a4adc7344d0b69eb3e3c8898f153eb4d"}
Oct 30 18:47:07.477869 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.477723    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" event={"ID":"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab","Type":"ContainerStarted","Data":"a2b72d11ea497ce692013440b6af20ea723ea2129733f432d3298af810f11828"}
Oct 30 18:47:07.925530 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.925483    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podStartSLOduration=1.925466957 podStartE2EDuration="1.925466957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:07.500479711 +0000 UTC m=+841.368898040" watchObservedRunningTime="2026-10-30 18:47:07.925466957 +0000 UTC m=+841.793885287"
Oct 30 18:47:07.925711 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.925609    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"]
Oct 30 18:47:07.929321 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.929303    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:07.940392 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.939401    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-aa6736\"/\"kube-root-ca.crt\""
Oct 30 18:47:07.940975 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.940951    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-aa6736\"/\"openshift-service-ca.crt\""
Oct 30 18:47:07.941201 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.941182    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-aa6736\"/\"distinct-2-w1-aa6736-kserve-self-signed-certs\""
Oct 30 18:47:07.941372 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.941328    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-aa6736\"/\"default-dockercfg-kzm7f\""
Oct 30 18:47:07.943499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.943475    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"]
Oct 30 18:47:07.992705 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.992681    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-tmp-dir\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:07.992797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.992715    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-dshm\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:07.992797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.992748    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dglx6\" (UniqueName: \"kubernetes.io/projected/b2b4e963-9553-43b4-81da-5787cdfd3753-kube-api-access-dglx6\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:07.992916 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.992844    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-home\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:07.992916 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.992885    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b4e963-9553-43b4-81da-5787cdfd3753-tls-certs\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:07.992980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:07.992912    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-model-cache\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:08.090153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:08.089626    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:08.093906 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:08.093862    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-model-cache\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:08.094078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:08.093987    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-tmp-dir\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:08.094078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:08.094026    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-dshm\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:08.094230 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:08.094078    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dglx6\" (UniqueName: \"kubernetes.io/projected/b2b4e963-9553-43b4-81da-5787cdfd3753-kube-api-access-dglx6\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:08.094230 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:08.094142    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-home\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:08.094230 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:08.094209    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b4e963-9553-43b4-81da-5787cdfd3753-tls-certs\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:08.095562 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:08.095205    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-home\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:08.095562 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:08.095406    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-model-cache\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:08.095562 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:08.095517    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-tmp-dir\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:08.104147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:08.102195    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-dshm\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:08.124501 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:08.119919    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dglx6\" (UniqueName: \"kubernetes.io/projected/b2b4e963-9553-43b4-81da-5787cdfd3753-kube-api-access-dglx6\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:08.124501 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:08.120786    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b4e963-9553-43b4-81da-5787cdfd3753-tls-certs\") pod \"distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:08.244922 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:08.244840    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:08.540429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:08.540402    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"]
Oct 30 18:47:08.541236 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:47:08.541212    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2b4e963_9553_43b4_81da_5787cdfd3753.slice/crio-4dcdab83f5db75dd7ed9f38fa0d1121902ba1df7c0fd749e8a107d90736c3ab3 WatchSource:0}: Error finding container 4dcdab83f5db75dd7ed9f38fa0d1121902ba1df7c0fd749e8a107d90736c3ab3: Status 404 returned error can't find the container with id 4dcdab83f5db75dd7ed9f38fa0d1121902ba1df7c0fd749e8a107d90736c3ab3
Oct 30 18:47:09.492292 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:09.492253    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" event={"ID":"b2b4e963-9553-43b4-81da-5787cdfd3753","Type":"ContainerStarted","Data":"49052149934e225e1ddba240f28f8f7183373c5b0525de129c01309d9d384087"}
Oct 30 18:47:09.492455 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:09.492311    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" event={"ID":"b2b4e963-9553-43b4-81da-5787cdfd3753","Type":"ContainerStarted","Data":"4dcdab83f5db75dd7ed9f38fa0d1121902ba1df7c0fd749e8a107d90736c3ab3"}
Oct 30 18:47:09.516111 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:09.516060    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podStartSLOduration=2.516046568 podStartE2EDuration="2.516046568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:09.513205205 +0000 UTC m=+843.381623536" watchObservedRunningTime="2026-10-30 18:47:09.516046568 +0000 UTC m=+843.384464899"
Oct 30 18:47:10.242544 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.242493    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56" podUID="94e81165-3be3-4050-bbf1-0452b618fbb2" containerName="main" containerID="cri-o://a4f80ce9e3c3c570b3ec56365ec61d52522f826327897b852ccf119f4903bb9b" gracePeriod=45
Oct 30 18:47:10.245972 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.245947    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:10.499555 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.499482    4372 generic.go:356] "Generic (PLEG): container finished" podID="94e81165-3be3-4050-bbf1-0452b618fbb2" containerID="a4f80ce9e3c3c570b3ec56365ec61d52522f826327897b852ccf119f4903bb9b" exitCode=0
Oct 30 18:47:10.499671 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.499563    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56" event={"ID":"94e81165-3be3-4050-bbf1-0452b618fbb2","Type":"ContainerDied","Data":"a4f80ce9e3c3c570b3ec56365ec61d52522f826327897b852ccf119f4903bb9b"}
Oct 30 18:47:10.499671 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.499604    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56" event={"ID":"94e81165-3be3-4050-bbf1-0452b618fbb2","Type":"ContainerDied","Data":"d23443d9ef043b392ed123dca0245ba7527e8aedb8365ca771985ecc201e6514"}
Oct 30 18:47:10.499671 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.499618    4372 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d23443d9ef043b392ed123dca0245ba7527e8aedb8365ca771985ecc201e6514"
Oct 30 18:47:10.508790 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.508773    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:47:10.622805 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.622765    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-dshm\") pod \"94e81165-3be3-4050-bbf1-0452b618fbb2\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") "
Oct 30 18:47:10.623010 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.622815    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-home\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-home\") pod \"94e81165-3be3-4050-bbf1-0452b618fbb2\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") "
Oct 30 18:47:10.623010 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.622870    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-tmp-dir\") pod \"94e81165-3be3-4050-bbf1-0452b618fbb2\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") "
Oct 30 18:47:10.623010 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.622907    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e81165-3be3-4050-bbf1-0452b618fbb2-kube-api-access-fjdfg\" (UniqueName: \"kubernetes.io/projected/94e81165-3be3-4050-bbf1-0452b618fbb2-kube-api-access-fjdfg\") pod \"94e81165-3be3-4050-bbf1-0452b618fbb2\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") "
Oct 30 18:47:10.623010 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.622991    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-model-cache\") pod \"94e81165-3be3-4050-bbf1-0452b618fbb2\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") "
Oct 30 18:47:10.623246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.623097    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94e81165-3be3-4050-bbf1-0452b618fbb2-tls-certs\" (UniqueName: \"kubernetes.io/secret/94e81165-3be3-4050-bbf1-0452b618fbb2-tls-certs\") pod \"94e81165-3be3-4050-bbf1-0452b618fbb2\" (UID: \"94e81165-3be3-4050-bbf1-0452b618fbb2\") "
Oct 30 18:47:10.623246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.623117    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-tmp-dir" pod "94e81165-3be3-4050-bbf1-0452b618fbb2" (UID: "94e81165-3be3-4050-bbf1-0452b618fbb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:47:10.623246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.623163    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-home" pod "94e81165-3be3-4050-bbf1-0452b618fbb2" (UID: "94e81165-3be3-4050-bbf1-0452b618fbb2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:47:10.623400 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.623340    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-model-cache" pod "94e81165-3be3-4050-bbf1-0452b618fbb2" (UID: "94e81165-3be3-4050-bbf1-0452b618fbb2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:47:10.623596 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.623574    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:10.623649 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.623600    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:10.623649 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.623616    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:10.625675 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.625641    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e81165-3be3-4050-bbf1-0452b618fbb2-kube-api-access-fjdfg" pod "94e81165-3be3-4050-bbf1-0452b618fbb2" (UID: "94e81165-3be3-4050-bbf1-0452b618fbb2"). InnerVolumeSpecName "kube-api-access-fjdfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:10.625782 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.625752    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-dshm" pod "94e81165-3be3-4050-bbf1-0452b618fbb2" (UID: "94e81165-3be3-4050-bbf1-0452b618fbb2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:47:10.625944 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.625915    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e81165-3be3-4050-bbf1-0452b618fbb2-tls-certs" pod "94e81165-3be3-4050-bbf1-0452b618fbb2" (UID: "94e81165-3be3-4050-bbf1-0452b618fbb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:47:10.724855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.724821    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e81165-3be3-4050-bbf1-0452b618fbb2-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:10.724855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.724851    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e81165-3be3-4050-bbf1-0452b618fbb2-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:10.725044 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:10.724868    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjdfg\" (UniqueName: \"kubernetes.io/projected/94e81165-3be3-4050-bbf1-0452b618fbb2-kube-api-access-fjdfg\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:11.028665 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.028634    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"]
Oct 30 18:47:11.029101 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.029085    4372 cpu_manager.go:414] "containerMap: removing container" podUID="94e81165-3be3-4050-bbf1-0452b618fbb2" containerName="main"
Oct 30 18:47:11.029101 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.029101    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e81165-3be3-4050-bbf1-0452b618fbb2" containerName="main"
Oct 30 18:47:11.029258 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.029183    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e81165-3be3-4050-bbf1-0452b618fbb2" containerName="main"
Oct 30 18:47:11.032416 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.032394    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.041045 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.041024    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-84c32a\"/\"kube-root-ca.crt\""
Oct 30 18:47:11.042581 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.042563    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-84c32a\"/\"default-dockercfg-7ql74\""
Oct 30 18:47:11.042581 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.042573    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-84c32a\"/\"distinct-w0-84c32a-kserve-self-signed-certs\""
Oct 30 18:47:11.043539 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.043527    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-84c32a\"/\"openshift-service-ca.crt\""
Oct 30 18:47:11.048277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.048254    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"]
Oct 30 18:47:11.130283 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.130257    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrww8\" (UniqueName: \"kubernetes.io/projected/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-kube-api-access-zrww8\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.130393 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.130369    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-model-cache\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.130435 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.130409    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-tmp-dir\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.130507 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.130495    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-dshm\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.130553 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.130542    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-home\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.130589 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.130566    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-tls-certs\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.232030 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.231982    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-model-cache\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.232405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.232375    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-tmp-dir\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.232615 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.232596    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-dshm\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.232775 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.232762    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-home\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.238030 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.232861    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-tls-certs\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.238356 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.238303    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrww8\" (UniqueName: \"kubernetes.io/projected/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-kube-api-access-zrww8\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.238636 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.236240    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-model-cache\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.238774 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.235901    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-home\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.251081 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.246688    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-tls-certs\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.251081 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.247209    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-dshm\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.251081 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.247275    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-tmp-dir\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.270976 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.269624    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrww8\" (UniqueName: \"kubernetes.io/projected/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-kube-api-access-zrww8\") pod \"distinct-w0-84c32a-kserve-df9987ccd-sk6b8\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.345181 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.345151    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:11.505662 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.505630    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"
Oct 30 18:47:11.538482 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.538446    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"]
Oct 30 18:47:11.548002 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.547979    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/e2e-conflict-a-89b77b0e-kserve-5cd8fff945-bvz56"]
Oct 30 18:47:11.671168 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:11.671143    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"]
Oct 30 18:47:11.671276 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:47:11.671142    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf72f91a_0fe0_46ee_8747_ee5e36e355d7.slice/crio-45cb257feb95a2abd8dac7c0b547548abacf8349125c7b32afb4437d5675595c WatchSource:0}: Error finding container 45cb257feb95a2abd8dac7c0b547548abacf8349125c7b32afb4437d5675595c: Status 404 returned error can't find the container with id 45cb257feb95a2abd8dac7c0b547548abacf8349125c7b32afb4437d5675595c
Oct 30 18:47:12.265026 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:12.264996    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e81165-3be3-4050-bbf1-0452b618fbb2" path="/var/lib/kubelet/pods/94e81165-3be3-4050-bbf1-0452b618fbb2/volumes"
Oct 30 18:47:12.514078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:12.514038    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" event={"ID":"cf72f91a-0fe0-46ee-8747-ee5e36e355d7","Type":"ContainerStarted","Data":"bc32abe46e8c9a378a22b70501b47407c913cd9a2d9ca41664c8feb3a0435a64"}
Oct 30 18:47:12.514078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:12.514085    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" event={"ID":"cf72f91a-0fe0-46ee-8747-ee5e36e355d7","Type":"ContainerStarted","Data":"45cb257feb95a2abd8dac7c0b547548abacf8349125c7b32afb4437d5675595c"}
Oct 30 18:47:12.541830 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:12.541719    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podStartSLOduration=2.541702886 podStartE2EDuration="2.541702886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:12.540420047 +0000 UTC m=+846.408838378" watchObservedRunningTime="2026-10-30 18:47:12.541702886 +0000 UTC m=+846.410121215"
Oct 30 18:47:13.345641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:13.345602    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:14.883154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:14.871896    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"]
Oct 30 18:47:14.905853 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:14.896415    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.033567 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.033519    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"]
Oct 30 18:47:15.091166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.051350    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sfn7\" (UniqueName: \"kubernetes.io/projected/f5de6051-31b1-4529-8db0-d6cb1320a743-kube-api-access-4sfn7\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.091166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.051419    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.091166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.051630    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.091166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.051726    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.091166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.051785    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.091166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.051964    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.091166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.052000    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.091166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.052056    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5de6051-31b1-4529-8db0-d6cb1320a743-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.091166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.052278    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.214858 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.214749    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.215142 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.215105    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.268493 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.268452    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.269556 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.223288    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.269733 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.222993    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.269872 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.269551    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.270072 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.270052    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5de6051-31b1-4529-8db0-d6cb1320a743-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.270292 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.270274    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.270504 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.270484    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfn7\" (UniqueName: \"kubernetes.io/projected/f5de6051-31b1-4529-8db0-d6cb1320a743-kube-api-access-4sfn7\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.271607 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.270688    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.271897 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.271881    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.293184 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.274776    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5de6051-31b1-4529-8db0-d6cb1320a743-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.293370 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.274862    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.293492 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.275517    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.486208 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.458957    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.486562 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.462864    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfn7\" (UniqueName: \"kubernetes.io/projected/f5de6051-31b1-4529-8db0-d6cb1320a743-kube-api-access-4sfn7\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.487142 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.487097    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.557154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.554149    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.658379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.658338    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:15.807204 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:15.797768    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:16.617231 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:16.617184    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"]
Oct 30 18:47:16.867368 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:16.867263    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:16.867649 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:16.867621    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:16.867882 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:16.867827    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:16.965603 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:16.965520    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-998xh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:47:16.965851 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:16.965821    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh" podUID="ccc2f38a-84e5-43c3-98d3-0f39ea658a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:17.089877 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.089842    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:17.179021 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.178558    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qstgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:47:17.179021 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.178618    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx" podUID="ce1f4775-99b1-4896-b58f-348c46e3b5f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:17.235063 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.235029    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"]
Oct 30 18:47:17.253047 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.253018    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.348430 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.348406    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"]
Oct 30 18:47:17.363274 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.362510    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:17.421202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.418766    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.421202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.418830    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh9vs\" (UniqueName: \"kubernetes.io/projected/a5fb1131-af1f-4ce3-8822-c305e2767c89-kube-api-access-wh9vs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.421202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.418909    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.421202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.418936    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.421202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.418998    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.421202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.419047    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.421202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.419095    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5fb1131-af1f-4ce3-8822-c305e2767c89-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.421202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.419144    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.421202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.419177    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.535150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.522813    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5fb1131-af1f-4ce3-8822-c305e2767c89-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.535150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.522878    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.535150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.522914    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.535150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.523030    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.535150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.523073    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh9vs\" (UniqueName: \"kubernetes.io/projected/a5fb1131-af1f-4ce3-8822-c305e2767c89-kube-api-access-wh9vs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.535150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.523163    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.535150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.523193    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.535150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.523258    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.535150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.523313    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.535150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.524820    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5fb1131-af1f-4ce3-8822-c305e2767c89-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.535150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.525082    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.535150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.525434    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.535150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.534445    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.535150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.534689    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.556649 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.556618    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.557671 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.557649    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.577420 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.577263    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh9vs\" (UniqueName: \"kubernetes.io/projected/a5fb1131-af1f-4ce3-8822-c305e2767c89-kube-api-access-wh9vs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.578084 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.578061    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.676168 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.676041    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:17.687938 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.687290    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54" event={"ID":"f5de6051-31b1-4529-8db0-d6cb1320a743","Type":"ContainerStarted","Data":"dbe1624a94eafa2e8363f87d6ef62d97bb15e1be8cfa1eaa7c113031f6d9e674"}
Oct 30 18:47:17.687938 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.687367    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54" event={"ID":"f5de6051-31b1-4529-8db0-d6cb1320a743","Type":"ContainerStarted","Data":"9754d265191d6061bb5b76d0a7d5c08530626625d9a805c60f2f73871daec623"}
Oct 30 18:47:17.711818 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.711657    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:17.725831 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.725753    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54" podStartSLOduration=3.72573257 podStartE2EDuration="3.72573257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:17.722285671 +0000 UTC m=+851.590704002" watchObservedRunningTime="2026-10-30 18:47:17.72573257 +0000 UTC m=+851.594150902"
Oct 30 18:47:17.985828 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.985803    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"]
Oct 30 18:47:17.986640 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:47:17.986614    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5fb1131_af1f_4ce3_8822_c305e2767c89.slice/crio-41c1fd5553dab09e0dc9392663a6a1e9361bc8b7f3fda19c2014a73dff932993 WatchSource:0}: Error finding container 41c1fd5553dab09e0dc9392663a6a1e9361bc8b7f3fda19c2014a73dff932993: Status 404 returned error can't find the container with id 41c1fd5553dab09e0dc9392663a6a1e9361bc8b7f3fda19c2014a73dff932993
Oct 30 18:47:17.989090 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.989052    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:17.989191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.989152    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:17.989191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:17.989187    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:18.257231 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:18.250547    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:18.272738 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:18.272691    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:18.659793 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:18.659759    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:18.661180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:18.661149    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 18:47:18.661337 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:18.661211    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54" podUID="f5de6051-31b1-4529-8db0-d6cb1320a743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 18:47:18.709743 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:18.709699    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q" event={"ID":"a5fb1131-af1f-4ce3-8822-c305e2767c89","Type":"ContainerStarted","Data":"c79682b4c5c91028a9edad9dea8569a5ba2276fb84b9e7572b3dda9e3a91889f"}
Oct 30 18:47:18.710276 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:18.709765    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q" event={"ID":"a5fb1131-af1f-4ce3-8822-c305e2767c89","Type":"ContainerStarted","Data":"41c1fd5553dab09e0dc9392663a6a1e9361bc8b7f3fda19c2014a73dff932993"}
Oct 30 18:47:18.721585 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:18.721551    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:18.819965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:18.819894    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q" podStartSLOduration=1.819869642 podStartE2EDuration="1.819869642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:18.757073109 +0000 UTC m=+852.625491441" watchObservedRunningTime="2026-10-30 18:47:18.819869642 +0000 UTC m=+852.688287973"
Oct 30 18:47:19.659289 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:19.659253    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 18:47:19.659482 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:19.659308    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54" podUID="f5de6051-31b1-4529-8db0-d6cb1320a743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 18:47:19.677183 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:19.677156    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:19.697342 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:19.697283    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:47:19.697669 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:19.697621    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q" podUID="a5fb1131-af1f-4ce3-8822-c305e2767c89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 18:47:20.661148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:20.659549    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 18:47:20.661148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:20.659624    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54" podUID="f5de6051-31b1-4529-8db0-d6cb1320a743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 18:47:20.680276 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:20.678764    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:47:20.680276 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:20.678831    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q" podUID="a5fb1131-af1f-4ce3-8822-c305e2767c89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 18:47:21.345565 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:21.345529    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:21.358601 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:21.358117    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:21.659882 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:21.659805    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 18:47:21.659882 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:21.659853    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54" podUID="f5de6051-31b1-4529-8db0-d6cb1320a743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 18:47:21.676934 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:21.676911    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:47:21.677306 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:21.676949    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q" podUID="a5fb1131-af1f-4ce3-8822-c305e2767c89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 18:47:21.932408 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:21.932280    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:22.659448 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:22.659415    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 18:47:22.659620 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:22.659480    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54" podUID="f5de6051-31b1-4529-8db0-d6cb1320a743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 18:47:22.677216 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:22.677191    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:47:22.677546 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:22.677245    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q" podUID="a5fb1131-af1f-4ce3-8822-c305e2767c89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 18:47:23.659962 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:23.659926    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 18:47:23.660170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:23.659975    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54" podUID="f5de6051-31b1-4529-8db0-d6cb1320a743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 18:47:23.677248 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:23.677225    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:47:23.677553 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:23.677264    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q" podUID="a5fb1131-af1f-4ce3-8822-c305e2767c89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 18:47:24.659325 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:24.659284    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 18:47:24.659479 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:24.659348    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54" podUID="f5de6051-31b1-4529-8db0-d6cb1320a743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 18:47:24.676661 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:24.676620    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:47:24.676870 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:24.676676    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q" podUID="a5fb1131-af1f-4ce3-8822-c305e2767c89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 18:47:24.901151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:24.872147    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"]
Oct 30 18:47:24.906470 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:24.906437    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"]
Oct 30 18:47:24.906943 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:24.906795    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.034632 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.034587    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.034632 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.034623    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fafb996-e969-4c99-be6b-421487aed44e-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.034912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.034647    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.034912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.034726    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fafb996-e969-4c99-be6b-421487aed44e-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.034912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.034796    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fafb996-e969-4c99-be6b-421487aed44e-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.034912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.034848    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97mqt\" (UniqueName: \"kubernetes.io/projected/9fafb996-e969-4c99-be6b-421487aed44e-kube-api-access-97mqt\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.035186 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.034962    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.035186 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.034983    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.035186 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.035038    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.136039 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.135992    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.136277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.136052    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.136277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.136186    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.136409 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.136329    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.136409 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.136356    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fafb996-e969-4c99-be6b-421487aed44e-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.136409 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.136395    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.136593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.136443    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fafb996-e969-4c99-be6b-421487aed44e-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.136593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.136469    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fafb996-e969-4c99-be6b-421487aed44e-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.136593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.136500    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97mqt\" (UniqueName: \"kubernetes.io/projected/9fafb996-e969-4c99-be6b-421487aed44e-kube-api-access-97mqt\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.136593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.136579    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.137436 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.137406    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.137833 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.137804    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.137948 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.137871    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.138087 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.138060    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fafb996-e969-4c99-be6b-421487aed44e-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.140493 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.140460    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.140898 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.140871    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fafb996-e969-4c99-be6b-421487aed44e-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.150920 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.150859    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97mqt\" (UniqueName: \"kubernetes.io/projected/9fafb996-e969-4c99-be6b-421487aed44e-kube-api-access-97mqt\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.151262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.151212    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fafb996-e969-4c99-be6b-421487aed44e-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.268798 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.268707    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:25.564288 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.564258    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"]
Oct 30 18:47:25.566798 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:47:25.566754    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fafb996_e969_4c99_be6b_421487aed44e.slice/crio-a0a89af93469ec296d9a5018b64962b420bdb0c43ea7040219ae8f977e424d1b WatchSource:0}: Error finding container a0a89af93469ec296d9a5018b64962b420bdb0c43ea7040219ae8f977e424d1b: Status 404 returned error can't find the container with id a0a89af93469ec296d9a5018b64962b420bdb0c43ea7040219ae8f977e424d1b
Oct 30 18:47:25.570000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.569964    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:25.570119 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.570044    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:25.570228 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.570114    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:25.664830 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.664801    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:25.665119 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.665095    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:25.666016 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.665998    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 18:47:25.681368 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.681304    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:25.681942 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.681870    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:25.682661 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.682634    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:47:25.890077 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.890038    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm" event={"ID":"9fafb996-e969-4c99-be6b-421487aed44e","Type":"ContainerStarted","Data":"64d8cfc1735e89a188a9a10309d67aafa966779130826b2e631db3df3581639d"}
Oct 30 18:47:25.890299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.890106    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm" event={"ID":"9fafb996-e969-4c99-be6b-421487aed44e","Type":"ContainerStarted","Data":"a0a89af93469ec296d9a5018b64962b420bdb0c43ea7040219ae8f977e424d1b"}
Oct 30 18:47:25.928347 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:25.928298    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm" podStartSLOduration=1.92828439 podStartE2EDuration="1.92828439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:25.9254162 +0000 UTC m=+859.793834531" watchObservedRunningTime="2026-10-30 18:47:25.92828439 +0000 UTC m=+859.796702720"
Oct 30 18:47:26.269904 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:26.269834    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:26.271258 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:26.271229    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 18:47:26.271415 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:26.271286    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm" podUID="9fafb996-e969-4c99-be6b-421487aed44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 18:47:27.270047 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:27.270009    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 18:47:27.270468 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:27.270077    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm" podUID="9fafb996-e969-4c99-be6b-421487aed44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 18:47:27.718068 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:47:27.718027    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:47:28.269083 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:28.269053    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 18:47:28.269299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:28.269109    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm" podUID="9fafb996-e969-4c99-be6b-421487aed44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 18:47:28.804112 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:28.804071    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:47:29.269230 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:29.269194    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 18:47:29.269376 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:29.269244    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm" podUID="9fafb996-e969-4c99-be6b-421487aed44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 18:47:30.281267 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:30.281228    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 18:47:30.281979 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:30.281940    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm" podUID="9fafb996-e969-4c99-be6b-421487aed44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 18:47:31.269979 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:31.269944    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 18:47:31.270182 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:31.270001    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm" podUID="9fafb996-e969-4c99-be6b-421487aed44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 18:47:32.269256 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.269224    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 18:47:32.269679 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.269284    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm" podUID="9fafb996-e969-4c99-be6b-421487aed44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 18:47:32.300798 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.300771    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"]
Oct 30 18:47:32.305194 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.305175    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.328043 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.328007    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"]
Oct 30 18:47:32.380732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.380699    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.380874 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.380837    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.380874 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.380866    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.380963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.380937    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.380963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.380957    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgmtd\" (UniqueName: \"kubernetes.io/projected/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-kube-api-access-zgmtd\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.381028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.380973    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.381151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.381112    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.381192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.381161    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.381192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.381183    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.481762 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.481735    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.481905 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.481771    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.482002 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.481980    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.482138 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.482106    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.482176 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.482151    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.482230 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.482184    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.482230 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.482208    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgmtd\" (UniqueName: \"kubernetes.io/projected/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-kube-api-access-zgmtd\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.482308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.482235    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.482308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.482242    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.482308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.482293    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.482469 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.482365    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.482567 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.482536    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.482715 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.482693    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.483048 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.483032    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.484702 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.484683    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.485321 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.485298    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.495465 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.495177    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgmtd\" (UniqueName: \"kubernetes.io/projected/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-kube-api-access-zgmtd\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.495465 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.495352    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.627664 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.627637    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:32.925623 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.925596    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"]
Oct 30 18:47:32.928112 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:47:32.928088    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10d45d9_3b28_4c0e_aeb6_62a63f39ab33.slice/crio-d8fe2f3d7981b289b7ee4081183557e70180e8322839fac39bf9a9a8fc940e7d WatchSource:0}: Error finding container d8fe2f3d7981b289b7ee4081183557e70180e8322839fac39bf9a9a8fc940e7d: Status 404 returned error can't find the container with id d8fe2f3d7981b289b7ee4081183557e70180e8322839fac39bf9a9a8fc940e7d
Oct 30 18:47:32.931484 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.931450    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:32.931575 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.931514    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:32.931575 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.931542    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:32.944984 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:32.944952    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw" event={"ID":"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33","Type":"ContainerStarted","Data":"d8fe2f3d7981b289b7ee4081183557e70180e8322839fac39bf9a9a8fc940e7d"}
Oct 30 18:47:33.358384 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:33.358350    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:33.358903 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:33.358802    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:33.360117 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:33.360095    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 18:47:33.952946 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:33.952903    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw" event={"ID":"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33","Type":"ContainerStarted","Data":"3ff76fe5c0ca7a230ee2d8aa1ba0a3fe1b54bb65abeaa1875b80c059a3d4db6f"}
Oct 30 18:47:34.004878 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:34.004793    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw" podStartSLOduration=2.004775252 podStartE2EDuration="2.004775252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:34.003013911 +0000 UTC m=+867.871432263" watchObservedRunningTime="2026-10-30 18:47:34.004775252 +0000 UTC m=+867.873193582"
Oct 30 18:47:34.628535 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:34.628502    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:34.630359 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:34.630324    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 18:47:34.630480 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:34.630380    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw" podUID="d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 18:47:35.629027 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:35.628993    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 18:47:35.629547 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:35.629051    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw" podUID="d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 18:47:36.548030 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.547991    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"]
Oct 30 18:47:36.552249 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.552216    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.560240 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.560164    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-home\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.560317 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.560241    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prh6n\" (UniqueName: \"kubernetes.io/projected/af1f5a15-28b6-497e-8c16-295736adf63a-kube-api-access-prh6n\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.560317 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.560282    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-tmp-dir\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.560434 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.560335    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1f5a15-28b6-497e-8c16-295736adf63a-tls-certs\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.560434 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.560383    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-model-cache\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.560434 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.560409    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-dshm\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.561194 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.561168    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-84c32a\"/\"distinct-2-w0-84c32a-kserve-self-signed-certs\""
Oct 30 18:47:36.564176 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.564150    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"]
Oct 30 18:47:36.628439 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.628409    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 18:47:36.628595 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.628462    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw" podUID="d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 18:47:36.661545 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.661515    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-dshm\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.661968 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.661684    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-home\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.661968 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.661724    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prh6n\" (UniqueName: \"kubernetes.io/projected/af1f5a15-28b6-497e-8c16-295736adf63a-kube-api-access-prh6n\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.661968 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.661762    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-tmp-dir\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.661968 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.661839    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1f5a15-28b6-497e-8c16-295736adf63a-tls-certs\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.661968 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.661901    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-model-cache\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.662235 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.662059    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-home\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.662235 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.662087    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-tmp-dir\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.662311 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.662231    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-model-cache\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.664025 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.664000    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-dshm\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.664368 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.664353    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1f5a15-28b6-497e-8c16-295736adf63a-tls-certs\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.737166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.737065    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prh6n\" (UniqueName: \"kubernetes.io/projected/af1f5a15-28b6-497e-8c16-295736adf63a-kube-api-access-prh6n\") pod \"distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") " pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:36.875357 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:36.874999    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:37.240416 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:37.240387    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"]
Oct 30 18:47:37.241711 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:47:37.241676    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf1f5a15_28b6_497e_8c16_295736adf63a.slice/crio-1c22720cf5af5e038420bd24d25e6653b90df7c21d9f7349ac4a663f24421a1e WatchSource:0}: Error finding container 1c22720cf5af5e038420bd24d25e6653b90df7c21d9f7349ac4a663f24421a1e: Status 404 returned error can't find the container with id 1c22720cf5af5e038420bd24d25e6653b90df7c21d9f7349ac4a663f24421a1e
Oct 30 18:47:37.628533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:37.628491    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 18:47:37.628734 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:37.628562    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw" podUID="d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 18:47:38.004167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:38.004044    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" event={"ID":"af1f5a15-28b6-497e-8c16-295736adf63a","Type":"ContainerStarted","Data":"76652f2e1acec527bdfb5fb62e69a941efbe5e34e4aecda1c125c34d8f24d1ff"}
Oct 30 18:47:38.004167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:38.004088    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" event={"ID":"af1f5a15-28b6-497e-8c16-295736adf63a","Type":"ContainerStarted","Data":"1c22720cf5af5e038420bd24d25e6653b90df7c21d9f7349ac4a663f24421a1e"}
Oct 30 18:47:38.024855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:38.024812    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podStartSLOduration=2.024799539 podStartE2EDuration="2.024799539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:38.023823958 +0000 UTC m=+871.892242285" watchObservedRunningTime="2026-10-30 18:47:38.024799539 +0000 UTC m=+871.893217902"
Oct 30 18:47:38.628855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:38.628818    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 18:47:38.629033 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:38.628869    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw" podUID="d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 18:47:38.876038 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:38.876009    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:39.628375 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:39.628343    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 18:47:39.628755 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:39.628391    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw" podUID="d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 18:47:40.632386 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:40.632361    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:40.632839 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:40.632784    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:40.633483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:40.633461    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 18:47:42.204152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.203320    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"]
Oct 30 18:47:42.231152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.228417    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.240933 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.240902    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"]
Oct 30 18:47:42.431421 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.431379    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09619584-4053-472a-9723-3dce678c9f4c-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.431696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.431671    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.431898 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.431876    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.432099 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.432080    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.432266 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.432248    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jncbz\" (UniqueName: \"kubernetes.io/projected/09619584-4053-472a-9723-3dce678c9f4c-kube-api-access-jncbz\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.432402 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.432386    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09619584-4053-472a-9723-3dce678c9f4c-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.432572 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.432556    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.432717 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.432702    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09619584-4053-472a-9723-3dce678c9f4c-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.434930 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.434906    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.552022 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.551915    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jncbz\" (UniqueName: \"kubernetes.io/projected/09619584-4053-472a-9723-3dce678c9f4c-kube-api-access-jncbz\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.552876 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.552850    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09619584-4053-472a-9723-3dce678c9f4c-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.607343 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.599819    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.635278 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.599922    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09619584-4053-472a-9723-3dce678c9f4c-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.635573 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.635545    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.635767 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.635748    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09619584-4053-472a-9723-3dce678c9f4c-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.640151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.635948    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.656243 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.640318    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.656648 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.656620    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.656895 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.553815    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09619584-4053-472a-9723-3dce678c9f4c-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.676181 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.643734    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.676441 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.644526    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.676746 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.634117    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jncbz\" (UniqueName: \"kubernetes.io/projected/09619584-4053-472a-9723-3dce678c9f4c-kube-api-access-jncbz\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.676873 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.657359    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.676993 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.657986    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09619584-4053-472a-9723-3dce678c9f4c-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.677139 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.672509    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.679300 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.679269    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:42.746182 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:42.744170    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09619584-4053-472a-9723-3dce678c9f4c-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:43.034354 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:43.034318    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:43.441154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:43.439747    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 18:47:43.522577 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:43.516147    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:43.693401 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:43.692212    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:43.773515 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:43.746939    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:44.094994 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:44.094947    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"]
Oct 30 18:47:44.398236 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:47:44.384351    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09619584_4053_472a_9723_3dce678c9f4c.slice/crio-70bda33ca8ef84215d6a44eb343d515d2888fb42ddbe19319eb0781cd5076c92 WatchSource:0}: Error finding container 70bda33ca8ef84215d6a44eb343d515d2888fb42ddbe19319eb0781cd5076c92: Status 404 returned error can't find the container with id 70bda33ca8ef84215d6a44eb343d515d2888fb42ddbe19319eb0781cd5076c92
Oct 30 18:47:44.441337 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:44.441284    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 18:47:44.445920 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:44.445894    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:47:44.457768 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:44.457713    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:44.458019 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:44.457992    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:44.458226 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:44.458202    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:44.532161 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:44.517587    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:44.544157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:44.533166    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:47:44.800205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:44.717998    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:44.800205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:44.718700    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:44.809158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:44.807845    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:44.809158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:44.808231    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:44.907152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:44.883864    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 18:47:45.130196 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:45.130168    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:47:45.468824 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:45.468784    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:45.514766 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:45.514298    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7" event={"ID":"09619584-4053-472a-9723-3dce678c9f4c","Type":"ContainerStarted","Data":"70bda33ca8ef84215d6a44eb343d515d2888fb42ddbe19319eb0781cd5076c92"}
Oct 30 18:47:45.542786 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:45.541224    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:45.850223 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:45.843413    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:45.850223 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:45.843877    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 30 18:47:45.893692 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:45.887769    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:47:45.895013 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:45.894224    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:46.000940 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:45.997032    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:47:46.002318 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:46.002114    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:47:46.161027 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:46.160948    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:47:46.257148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:46.241730    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:47:46.651338 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:46.651245    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7" event={"ID":"09619584-4053-472a-9723-3dce678c9f4c","Type":"ContainerStarted","Data":"e72e71e827125b14124aa108ec9c48dfed1e0869f0d39bfd2171f7681908fcca"}
Oct 30 18:47:46.681190 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:46.681098    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7" podStartSLOduration=4.681078821 podStartE2EDuration="4.681078821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:46.677358814 +0000 UTC m=+880.545777141" watchObservedRunningTime="2026-10-30 18:47:46.681078821 +0000 UTC m=+880.549497151"
Oct 30 18:47:46.876067 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:46.876036    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:46.889888 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:46.889868    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:47.035320 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:47.035254    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:47.036302 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:47.036281    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:47:47.036412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:47.036329    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7" podUID="09619584-4053-472a-9723-3dce678c9f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 18:47:47.668498 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:47.668465    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:47:48.046203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:48.037542    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:47:48.046203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:48.037604    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7" podUID="09619584-4053-472a-9723-3dce678c9f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 18:47:49.035768 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:49.035736    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:47:49.036217 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:49.035789    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7" podUID="09619584-4053-472a-9723-3dce678c9f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 18:47:50.035784 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:50.035750    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:47:50.036212 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:50.035800    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7" podUID="09619584-4053-472a-9723-3dce678c9f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 18:47:51.035895 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:51.035863    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:47:51.036321 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:51.035917    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7" podUID="09619584-4053-472a-9723-3dce678c9f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 18:47:52.035709 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:52.035674    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:47:52.035897 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:52.035724    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7" podUID="09619584-4053-472a-9723-3dce678c9f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 18:47:53.035105 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:53.035063    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:47:53.035517 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:53.035195    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7" podUID="09619584-4053-472a-9723-3dce678c9f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 18:47:54.039257 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:54.039229    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:54.039715 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:54.039507    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:54.040338 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:54.040318    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:47:57.695046 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:47:57.695012    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:47:59.064876 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:47:59.064835    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:48:01.665060 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:01.664271    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:01.851013 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:01.850966    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:02.016506 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.016410    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 18:48:02.291215 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.243742    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 18:48:02.291215 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.252207    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:02.371153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.367620    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:02.486177 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.486145    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"]
Oct 30 18:48:02.513771 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.513742    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"]
Oct 30 18:48:02.514159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.514071    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.594273 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.594222    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:02.665820 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.665777    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:02.687285 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.687256    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:48:02.696156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.691722    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.696411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.696383    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5567ca68-ab6f-496b-8538-c6102a04364c-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.697158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.696536    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5567ca68-ab6f-496b-8538-c6102a04364c-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.697340 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.697318    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.701493 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.701469    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlc6t\" (UniqueName: \"kubernetes.io/projected/5567ca68-ab6f-496b-8538-c6102a04364c-kube-api-access-hlc6t\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.701720 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.701700    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.704163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.701864    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5567ca68-ab6f-496b-8538-c6102a04364c-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.704369 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.704344    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.704625 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.704604    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.849734 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.849681    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.850153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.850114    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.850444 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.850412    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.850606 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.850587    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5567ca68-ab6f-496b-8538-c6102a04364c-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.850762 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.850744    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5567ca68-ab6f-496b-8538-c6102a04364c-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.850912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.850895    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.851106 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.851071    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlc6t\" (UniqueName: \"kubernetes.io/projected/5567ca68-ab6f-496b-8538-c6102a04364c-kube-api-access-hlc6t\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.872182 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.863687    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.872182 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.863750    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5567ca68-ab6f-496b-8538-c6102a04364c-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.872182 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.864903    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5567ca68-ab6f-496b-8538-c6102a04364c-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.917177 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.917145    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.917427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.852647    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:02.917631 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.917581    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:02.918578 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.918559    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:48:02.957153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.930683    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:03.015384 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.853692    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:03.015601 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.965993    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlc6t\" (UniqueName: \"kubernetes.io/projected/5567ca68-ab6f-496b-8538-c6102a04364c-kube-api-access-hlc6t\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:03.015706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:02.984997    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:03.015851 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:03.015823    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:03.077636 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:03.032102    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:03.077636 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:03.043703    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5567ca68-ab6f-496b-8538-c6102a04364c-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:03.077636 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:03.066741    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:48:03.197216 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:03.180531    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:03.333157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:03.332400    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 18:48:03.366152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:03.351268    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:03.366152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:03.354334    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:48:03.366152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:03.354441    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:48:03.366152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:03.363311    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5567ca68-ab6f-496b-8538-c6102a04364c-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:03.562963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:03.562881    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:48:03.581425 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:03.579281    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:03.602293 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:03.602263    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:48:03.732873 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:03.732847    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:48:03.928232 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:03.925263    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:04.038698 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:04.036874    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:04.074188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:04.073891    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:48:04.129744 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:04.129721    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:48:04.370760 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:04.370732    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"]
Oct 30 18:48:04.373240 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:48:04.373207    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5567ca68_ab6f_496b_8538_c6102a04364c.slice/crio-62c7dc6062fc7ba29a0f8bd6b3c4d149ef51326890dedac80ea27b068aba9072 WatchSource:0}: Error finding container 62c7dc6062fc7ba29a0f8bd6b3c4d149ef51326890dedac80ea27b068aba9072: Status 404 returned error can't find the container with id 62c7dc6062fc7ba29a0f8bd6b3c4d149ef51326890dedac80ea27b068aba9072
Oct 30 18:48:04.375512 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:04.375477    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:04.375627 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:04.375542    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:04.375627 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:04.375574    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:05.070800 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:05.070741    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t" event={"ID":"5567ca68-ab6f-496b-8538-c6102a04364c","Type":"ContainerStarted","Data":"bcd8f974ac4b3ffa15650dcd2d8ab25ce111b215a617713ba0d3dcee33ef400b"}
Oct 30 18:48:05.071358 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:05.070845    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t" event={"ID":"5567ca68-ab6f-496b-8538-c6102a04364c","Type":"ContainerStarted","Data":"62c7dc6062fc7ba29a0f8bd6b3c4d149ef51326890dedac80ea27b068aba9072"}
Oct 30 18:48:05.146329 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:05.146272    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t" podStartSLOduration=3.146259114 podStartE2EDuration="3.146259114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:05.144943139 +0000 UTC m=+899.013361492" watchObservedRunningTime="2026-10-30 18:48:05.146259114 +0000 UTC m=+899.014677443"
Oct 30 18:48:05.604699 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:05.604661    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:05.605992 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:05.605963    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 18:48:05.606149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:05.606012    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t" podUID="5567ca68-ab6f-496b-8538-c6102a04364c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 18:48:06.604491 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:06.604454    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 18:48:06.605027 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:06.604519    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t" podUID="5567ca68-ab6f-496b-8538-c6102a04364c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 18:48:07.605288 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:07.605256    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 18:48:07.605701 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:07.605309    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t" podUID="5567ca68-ab6f-496b-8538-c6102a04364c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 18:48:07.859625 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:07.859550    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"]
Oct 30 18:48:07.864839 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:07.864820    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"
Oct 30 18:48:07.870502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:07.870480    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:48:07.870741 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:07.870725    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 18:48:07.877092 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:07.877071    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"]
Oct 30 18:48:07.913086 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:07.913055    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10da9fc4-c422-4802-96e4-222a3274b87d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0362f6d7-55jjw\" (UID: \"10da9fc4-c422-4802-96e4-222a3274b87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"
Oct 30 18:48:07.913249 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:07.913095    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10da9fc4-c422-4802-96e4-222a3274b87d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0362f6d7-55jjw\" (UID: \"10da9fc4-c422-4802-96e4-222a3274b87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"
Oct 30 18:48:07.913434 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:07.913404    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brz5f\" (UniqueName: \"kubernetes.io/projected/10da9fc4-c422-4802-96e4-222a3274b87d-kube-api-access-brz5f\") pod \"maas-api-revoke-sub-0362f6d7-55jjw\" (UID: \"10da9fc4-c422-4802-96e4-222a3274b87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"
Oct 30 18:48:08.014887 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:08.014848    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brz5f\" (UniqueName: \"kubernetes.io/projected/10da9fc4-c422-4802-96e4-222a3274b87d-kube-api-access-brz5f\") pod \"maas-api-revoke-sub-0362f6d7-55jjw\" (UID: \"10da9fc4-c422-4802-96e4-222a3274b87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"
Oct 30 18:48:08.015040 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:08.014925    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10da9fc4-c422-4802-96e4-222a3274b87d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0362f6d7-55jjw\" (UID: \"10da9fc4-c422-4802-96e4-222a3274b87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"
Oct 30 18:48:08.015040 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:08.014955    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10da9fc4-c422-4802-96e4-222a3274b87d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0362f6d7-55jjw\" (UID: \"10da9fc4-c422-4802-96e4-222a3274b87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"
Oct 30 18:48:08.015652 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:08.015626    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10da9fc4-c422-4802-96e4-222a3274b87d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0362f6d7-55jjw\" (UID: \"10da9fc4-c422-4802-96e4-222a3274b87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"
Oct 30 18:48:08.017479 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:08.017460    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10da9fc4-c422-4802-96e4-222a3274b87d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0362f6d7-55jjw\" (UID: \"10da9fc4-c422-4802-96e4-222a3274b87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"
Oct 30 18:48:08.030994 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:08.030968    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brz5f\" (UniqueName: \"kubernetes.io/projected/10da9fc4-c422-4802-96e4-222a3274b87d-kube-api-access-brz5f\") pod \"maas-api-revoke-sub-0362f6d7-55jjw\" (UID: \"10da9fc4-c422-4802-96e4-222a3274b87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"
Oct 30 18:48:08.205459 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:08.205377    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"
Oct 30 18:48:08.504561 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:08.504532    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"]
Oct 30 18:48:08.509386 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:48:08.509351    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10da9fc4_c422_4802_96e4_222a3274b87d.slice/crio-8f15861025a0409516d1509b4ec7cd11f5e488c8754b9f9cfe30b65d6ebed62b WatchSource:0}: Error finding container 8f15861025a0409516d1509b4ec7cd11f5e488c8754b9f9cfe30b65d6ebed62b: Status 404 returned error can't find the container with id 8f15861025a0409516d1509b4ec7cd11f5e488c8754b9f9cfe30b65d6ebed62b
Oct 30 18:48:08.606014 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:08.605984    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 18:48:08.606460 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:08.606039    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t" podUID="5567ca68-ab6f-496b-8538-c6102a04364c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 18:48:09.116348 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:09.116313    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw" event={"ID":"10da9fc4-c422-4802-96e4-222a3274b87d","Type":"ContainerStarted","Data":"dab16fb914a8a07190bafcb38a2ff335b656e561bcf4e89465d0df530e457141"}
Oct 30 18:48:09.116525 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:09.116373    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw" event={"ID":"10da9fc4-c422-4802-96e4-222a3274b87d","Type":"ContainerStarted","Data":"8f15861025a0409516d1509b4ec7cd11f5e488c8754b9f9cfe30b65d6ebed62b"}
Oct 30 18:48:09.145031 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:09.144984    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw" podStartSLOduration=2.144968948 podStartE2EDuration="2.144968948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:09.139574221 +0000 UTC m=+903.007992552" watchObservedRunningTime="2026-10-30 18:48:09.144968948 +0000 UTC m=+903.013387278"
Oct 30 18:48:09.614157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:09.612540    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 18:48:09.614157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:09.612605    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t" podUID="5567ca68-ab6f-496b-8538-c6102a04364c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 18:48:09.999163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:09.978443    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"]
Oct 30 18:48:10.009167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.003551    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.014148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.011234    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"]
Oct 30 18:48:10.045035 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.045005    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/450e8016-b275-4172-ae20-5668c854e43a-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.045197 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.045041    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.045197 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.045065    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.045280 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.045214    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk86l\" (UniqueName: \"kubernetes.io/projected/450e8016-b275-4172-ae20-5668c854e43a-kube-api-access-hk86l\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.045280 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.045253    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.045280 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.045279    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/450e8016-b275-4172-ae20-5668c854e43a-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.045408 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.045396    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/450e8016-b275-4172-ae20-5668c854e43a-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.045507 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.045494    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.045557 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.045524    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.124540 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.124504    4372 generic.go:356] "Generic (PLEG): container finished" podID="10da9fc4-c422-4802-96e4-222a3274b87d" containerID="dab16fb914a8a07190bafcb38a2ff335b656e561bcf4e89465d0df530e457141" exitCode=0
Oct 30 18:48:10.124683 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.124564    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw" event={"ID":"10da9fc4-c422-4802-96e4-222a3274b87d","Type":"ContainerDied","Data":"dab16fb914a8a07190bafcb38a2ff335b656e561bcf4e89465d0df530e457141"}
Oct 30 18:48:10.146777 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.146740    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/450e8016-b275-4172-ae20-5668c854e43a-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.146912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.146789    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.146912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.146833    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.146912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.146895    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk86l\" (UniqueName: \"kubernetes.io/projected/450e8016-b275-4172-ae20-5668c854e43a-kube-api-access-hk86l\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.147083 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.146940    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.147083 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.146977    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/450e8016-b275-4172-ae20-5668c854e43a-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.147083 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.147025    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/450e8016-b275-4172-ae20-5668c854e43a-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.147252 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.147174    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.147252 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.147223    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.147361 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.147301    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.147361 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.147342    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.147566 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.147549    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.147651 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.147627    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.147758 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.147743    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/450e8016-b275-4172-ae20-5668c854e43a-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.150192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.150171    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.150465 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.150441    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/450e8016-b275-4172-ae20-5668c854e43a-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.161798 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.161777    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/450e8016-b275-4172-ae20-5668c854e43a-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.162500 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.162475    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk86l\" (UniqueName: \"kubernetes.io/projected/450e8016-b275-4172-ae20-5668c854e43a-kube-api-access-hk86l\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.324369 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.324337    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:10.604480 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.604406    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 18:48:10.604480 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.604455    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t" podUID="5567ca68-ab6f-496b-8538-c6102a04364c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 18:48:10.635921 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.635884    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"]
Oct 30 18:48:10.636697 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:48:10.636653    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450e8016_b275_4172_ae20_5668c854e43a.slice/crio-d74ced4bee3dc6657ef2a5ea50870f888f66053515993e1371817ebc33825d70 WatchSource:0}: Error finding container d74ced4bee3dc6657ef2a5ea50870f888f66053515993e1371817ebc33825d70: Status 404 returned error can't find the container with id d74ced4bee3dc6657ef2a5ea50870f888f66053515993e1371817ebc33825d70
Oct 30 18:48:10.639182 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.639120    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:10.639262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.639212    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:10.639262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:10.639241    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:11.135679 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.135643    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" event={"ID":"450e8016-b275-4172-ae20-5668c854e43a","Type":"ContainerStarted","Data":"75ffe7ae3d272a243f1e65521b236a993a45f30dc4ed3ff5ac94417e214d988d"}
Oct 30 18:48:11.135865 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.135717    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" event={"ID":"450e8016-b275-4172-ae20-5668c854e43a","Type":"ContainerStarted","Data":"d74ced4bee3dc6657ef2a5ea50870f888f66053515993e1371817ebc33825d70"}
Oct 30 18:48:11.181638 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.181584    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" podStartSLOduration=2.181566847 podStartE2EDuration="2.181566847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:11.17949047 +0000 UTC m=+905.047908800" watchObservedRunningTime="2026-10-30 18:48:11.181566847 +0000 UTC m=+905.049985177"
Oct 30 18:48:11.313639 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.313608    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"
Oct 30 18:48:11.325540 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.325446    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:11.328961 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.328931    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 18:48:11.329199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.329173    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" podUID="450e8016-b275-4172-ae20-5668c854e43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 18:48:11.385733 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.385634    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10da9fc4-c422-4802-96e4-222a3274b87d-kube-api-access-brz5f\" (UniqueName: \"kubernetes.io/projected/10da9fc4-c422-4802-96e4-222a3274b87d-kube-api-access-brz5f\") pod \"10da9fc4-c422-4802-96e4-222a3274b87d\" (UID: \"10da9fc4-c422-4802-96e4-222a3274b87d\") "
Oct 30 18:48:11.385902 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.385754    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10da9fc4-c422-4802-96e4-222a3274b87d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10da9fc4-c422-4802-96e4-222a3274b87d-maas-api-xks-ca\") pod \"10da9fc4-c422-4802-96e4-222a3274b87d\" (UID: \"10da9fc4-c422-4802-96e4-222a3274b87d\") "
Oct 30 18:48:11.385902 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.385797    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10da9fc4-c422-4802-96e4-222a3274b87d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10da9fc4-c422-4802-96e4-222a3274b87d-maas-api-openshift-ca\") pod \"10da9fc4-c422-4802-96e4-222a3274b87d\" (UID: \"10da9fc4-c422-4802-96e4-222a3274b87d\") "
Oct 30 18:48:11.407929 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.394999    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10da9fc4-c422-4802-96e4-222a3274b87d-maas-api-openshift-ca" pod "10da9fc4-c422-4802-96e4-222a3274b87d" (UID: "10da9fc4-c422-4802-96e4-222a3274b87d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:11.407929 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.398863    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10da9fc4-c422-4802-96e4-222a3274b87d-kube-api-access-brz5f" pod "10da9fc4-c422-4802-96e4-222a3274b87d" (UID: "10da9fc4-c422-4802-96e4-222a3274b87d"). InnerVolumeSpecName "kube-api-access-brz5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:11.424929 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.424652    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10da9fc4-c422-4802-96e4-222a3274b87d-maas-api-xks-ca" pod "10da9fc4-c422-4802-96e4-222a3274b87d" (UID: "10da9fc4-c422-4802-96e4-222a3274b87d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:11.491150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.489241    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brz5f\" (UniqueName: \"kubernetes.io/projected/10da9fc4-c422-4802-96e4-222a3274b87d-kube-api-access-brz5f\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:11.491150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.489283    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10da9fc4-c422-4802-96e4-222a3274b87d-maas-api-xks-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:11.491150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.489297    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10da9fc4-c422-4802-96e4-222a3274b87d-maas-api-openshift-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:11.865434 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.865403    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:11.865969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.865934    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:11.866859 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:11.866838    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:48:12.143844 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.143769    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"
Oct 30 18:48:12.143844 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.143803    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw" event={"ID":"10da9fc4-c422-4802-96e4-222a3274b87d","Type":"ContainerDied","Data":"8f15861025a0409516d1509b4ec7cd11f5e488c8754b9f9cfe30b65d6ebed62b"}
Oct 30 18:48:12.143844 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.143848    4372 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f15861025a0409516d1509b4ec7cd11f5e488c8754b9f9cfe30b65d6ebed62b"
Oct 30 18:48:12.272069 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.272032    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"]
Oct 30 18:48:12.272583 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.272569    4372 cpu_manager.go:414] "containerMap: removing container" podUID="10da9fc4-c422-4802-96e4-222a3274b87d" containerName="revoke-keys"
Oct 30 18:48:12.272634 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.272585    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10da9fc4-c422-4802-96e4-222a3274b87d" containerName="revoke-keys"
Oct 30 18:48:12.272681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.272661    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="10da9fc4-c422-4802-96e4-222a3274b87d" containerName="revoke-keys"
Oct 30 18:48:12.276032 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.276015    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.311651 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.311614    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"]
Oct 30 18:48:12.325522 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.325497    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 18:48:12.325636 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.325542    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" podUID="450e8016-b275-4172-ae20-5668c854e43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 18:48:12.420963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.420882    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.420963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.420916    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.420963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.420939    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnkf6\" (UniqueName: \"kubernetes.io/projected/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-kube-api-access-cnkf6\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.421228 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.421031    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.421228 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.421066    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.421228 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.421217    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.421343 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.421308    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.421395 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.421382    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.421436 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.421428    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.522498 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.522462    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.522681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.522546    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.522681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.522595    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.522681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.522636    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.522681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.522667    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.522859 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.522683    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.522859 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.522704    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnkf6\" (UniqueName: \"kubernetes.io/projected/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-kube-api-access-cnkf6\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.522859 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.522740    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.522859 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.522770    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.522995 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.522905    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.523046 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.523017    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.523278 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.523258    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.523359 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.523344    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.523502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.523481    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.525361 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.525341    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.525631 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.525605    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.537060 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.537040    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnkf6\" (UniqueName: \"kubernetes.io/projected/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-kube-api-access-cnkf6\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.543474 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.543451    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:12.630154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:12.626304    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:13.031501 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:48:13.031466    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd3fc6c_882e_4db7_a2b6_6886cb3bfc5c.slice/crio-34a9378bbb89c81fc7adc3f150196243964cc4ef3cc5ede807cf1f93be8ba4b2 WatchSource:0}: Error finding container 34a9378bbb89c81fc7adc3f150196243964cc4ef3cc5ede807cf1f93be8ba4b2: Status 404 returned error can't find the container with id 34a9378bbb89c81fc7adc3f150196243964cc4ef3cc5ede807cf1f93be8ba4b2
Oct 30 18:48:13.032505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:13.032483    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"]
Oct 30 18:48:13.034405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:13.034376    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:13.034460 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:13.034437    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:13.034500 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:13.034468    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:13.154046 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:13.154004    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" event={"ID":"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c","Type":"ContainerStarted","Data":"8a096cb4590e3de82bd3c9185343f0edc131a2c8a682eaf9268da1d60cc53838"}
Oct 30 18:48:13.154250 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:13.154109    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" event={"ID":"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c","Type":"ContainerStarted","Data":"34a9378bbb89c81fc7adc3f150196243964cc4ef3cc5ede807cf1f93be8ba4b2"}
Oct 30 18:48:13.214165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:13.213579    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" podStartSLOduration=1.213558432 podStartE2EDuration="1.213558432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:13.182678139 +0000 UTC m=+907.051096472" watchObservedRunningTime="2026-10-30 18:48:13.213558432 +0000 UTC m=+907.081976763"
Oct 30 18:48:13.324849 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:13.324809    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 18:48:13.325033 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:13.324889    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" podUID="450e8016-b275-4172-ae20-5668c854e43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 18:48:13.650236 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:13.642238    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:14.329549 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:14.329510    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 18:48:14.330162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:14.330104    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" podUID="450e8016-b275-4172-ae20-5668c854e43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 18:48:14.645216 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:14.644535    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 18:48:14.645216 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:14.644611    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" podUID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 18:48:15.132250 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.132215    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"]
Oct 30 18:48:15.370178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.352588    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 18:48:15.370178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.352651    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" podUID="450e8016-b275-4172-ae20-5668c854e43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 18:48:15.371995 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.371970    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.408202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.407319    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2aada387-openshift-default-dockercfg-4srp6\""
Oct 30 18:48:15.451192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.436705    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-credential-socket\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.458450 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.458406    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctsqb\" (UniqueName: \"kubernetes.io/projected/caf281ca-692b-48c8-84d7-13ca8e6e7489-kube-api-access-ctsqb\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.458687 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.458665    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-podinfo\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.458867 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.458849    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-workload-certs\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.468150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.459004    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-data\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.509584 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.509553    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-token\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.509951 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.509911    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-workload-socket\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.510097 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.510080    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-envoy\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.534740 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.534706    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf281ca-692b-48c8-84d7-13ca8e6e7489-istiod-ca-cert\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.561988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.371835    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"]
Oct 30 18:48:15.607976 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.607934    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 18:48:15.647265 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.647229    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 18:48:15.647509 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.647483    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" podUID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 18:48:15.647667 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.647271    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-token\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.647854 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.647835    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-workload-socket\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.647984 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.647968    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-envoy\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.648087 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.648072    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf281ca-692b-48c8-84d7-13ca8e6e7489-istiod-ca-cert\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.652153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.648303    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-credential-socket\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.652345 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.652322    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctsqb\" (UniqueName: \"kubernetes.io/projected/caf281ca-692b-48c8-84d7-13ca8e6e7489-kube-api-access-ctsqb\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.677570 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.677499    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-podinfo\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.678244 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.678223    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-workload-certs\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.678795 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.678777    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-data\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.679397 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.675404    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-credential-socket\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.679530 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.676333    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf281ca-692b-48c8-84d7-13ca8e6e7489-istiod-ca-cert\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.679629 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.678695    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-workload-certs\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.679741 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.677421    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-workload-socket\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.679838 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.679179    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-data\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.717818 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.717783    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-podinfo\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.724201 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.719676    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-token\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.724201 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.723326    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-envoy\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.773073 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.755507    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:15.773073 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.762179    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctsqb\" (UniqueName: \"kubernetes.io/projected/caf281ca-692b-48c8-84d7-13ca8e6e7489-kube-api-access-ctsqb\") pod \"e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") " pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.791991 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.790810    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:15.794490 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:15.793019    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:16.078259 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:16.078212    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:16.137152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:16.133716    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:16.338966 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:16.338910    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 18:48:16.352153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:16.346209    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" podUID="450e8016-b275-4172-ae20-5668c854e43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 18:48:16.560308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:16.555290    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 18:48:16.612203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:16.608546    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 18:48:16.767151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:16.742732    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:48:16.767151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:16.619644    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 18:48:16.767151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:16.645615    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 18:48:16.767151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:16.744396    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" podUID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 18:48:16.767151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:16.763857    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:16.767151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:16.764179    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:48:17.086282 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.086234    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:17.102536 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.102503    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:48:17.158745 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.158698    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:17.159276 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.159226    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:48:17.421165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.397176    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:17.421165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.397278    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:17.421165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.397329    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:17.440178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.438669    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"]
Oct 30 18:48:17.440178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.438789    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:17.540406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.540371    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 18:48:17.550990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.550946    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" podUID="450e8016-b275-4172-ae20-5668c854e43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 18:48:17.632002 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.609386    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 18:48:17.632837 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.629791    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 18:48:17.633016 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.632990    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" podUID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 18:48:17.633215 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.633198    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:48:17.827175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.780313    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:17.827175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.783208    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:17.827175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.783545    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:17.827175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.783900    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:48:17.938296 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:17.921372    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:18.070443 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:18.068478    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:18.070443 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:18.068564    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:18.153998 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:18.153958    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:18.411150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:18.398011    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 18:48:18.783763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:18.783715    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:18.886768 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:18.833907    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 18:48:18.915538 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:18.915502    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 18:48:19.071201 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.068737    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" podUID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 18:48:19.071201 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.031759    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:19.241151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.083318    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:19.272422 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.272388    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 18:48:19.272672 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.272645    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" podUID="450e8016-b275-4172-ae20-5668c854e43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 18:48:19.327270 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.326634    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:48:19.327270 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.326692    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"]
Oct 30 18:48:19.334684 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.334651    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 18:48:19.338947 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.338918    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 18:48:19.339183 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.339142    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" podUID="450e8016-b275-4172-ae20-5668c854e43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 18:48:19.361342 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.356299    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf" event={"ID":"caf281ca-692b-48c8-84d7-13ca8e6e7489","Type":"ContainerStarted","Data":"5226bd71780cbcbb8bb6fb9634ca9d5f2663b6935b7933f628d225cdeca9490b"}
Oct 30 18:48:19.361342 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.356345    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"]
Oct 30 18:48:19.361342 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.356412    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:48:19.361342 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.356444    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:48:19.361342 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.356945    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.361342 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.359373    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:48:19.374523 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.371672    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:48:19.374523 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.371730    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:48:19.438562 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.438507    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf" podStartSLOduration=4.438485717 podStartE2EDuration="4.438485717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:19.429616673 +0000 UTC m=+913.298035044" watchObservedRunningTime="2026-10-30 18:48:19.438485717 +0000 UTC m=+913.306904050"
Oct 30 18:48:19.474833 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.474796    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.475082 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.475060    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.475418 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.475220    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnmfd\" (UniqueName: \"kubernetes.io/projected/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-kube-api-access-lnmfd\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.475418 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.475355    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.475619 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.475442    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.475619 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.475535    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.475619 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.475575    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.475843 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.475822    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.476034 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.476016    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.577732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.577568    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.577732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.577617    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.577732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.577649    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmfd\" (UniqueName: \"kubernetes.io/projected/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-kube-api-access-lnmfd\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.577732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.577684    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.577732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.577723    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.578162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.577770    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.578162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.577799    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.578162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.577852    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.578162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.577869    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.599150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.578709    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.599150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.592353    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.599150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.592618    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.599482 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.599456    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.600777 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.600751    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.627802 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.627773    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.629443 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.629417    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.629583 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.629554    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.629677 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.629651    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 18:48:19.629739 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.629689    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" podUID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 18:48:19.652174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.650006    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnmfd\" (UniqueName: \"kubernetes.io/projected/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-kube-api-access-lnmfd\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.711501 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.711205    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:19.799376 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:19.799215    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:20.211904 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:48:20.211858    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba48ce60_30f0_4bb7_8d76_bbb4a32ae284.slice/crio-3a4aa3ef8db54a929abdf35df501d977aa85a2dc34477305b63a6cdf04da73f0 WatchSource:0}: Error finding container 3a4aa3ef8db54a929abdf35df501d977aa85a2dc34477305b63a6cdf04da73f0: Status 404 returned error can't find the container with id 3a4aa3ef8db54a929abdf35df501d977aa85a2dc34477305b63a6cdf04da73f0
Oct 30 18:48:20.212568 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:20.212538    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"]
Oct 30 18:48:20.214751 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:20.214362    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:20.214751 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:20.214429    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:20.214751 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:20.214459    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:20.325290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:20.325250    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 18:48:20.325399 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:20.325305    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" podUID="450e8016-b275-4172-ae20-5668c854e43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 18:48:20.347608 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:20.347377    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv" event={"ID":"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284","Type":"ContainerStarted","Data":"2603f53f28a8a57885876aba10a64ff5c8f3ba41451b81f3ee2bbdb92b405e4a"}
Oct 30 18:48:20.347608 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:20.347454    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv" event={"ID":"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284","Type":"ContainerStarted","Data":"3a4aa3ef8db54a929abdf35df501d977aa85a2dc34477305b63a6cdf04da73f0"}
Oct 30 18:48:20.350347 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:20.350320    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf" event={"ID":"caf281ca-692b-48c8-84d7-13ca8e6e7489","Type":"ContainerStarted","Data":"903c5942727be773f1336a3b5866d3e00b05992470e4bc2466b3f43868ef57af"}
Oct 30 18:48:20.630945 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:20.630895    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 18:48:20.632671 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:20.632629    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" podUID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 18:48:20.712271 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:20.712234    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:20.806426 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:20.806392    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:20.840894 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:20.839904    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv" podStartSLOduration=3.839882169 podStartE2EDuration="3.839882169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:20.417404909 +0000 UTC m=+914.285823288" watchObservedRunningTime="2026-10-30 18:48:20.839882169 +0000 UTC m=+914.708300499"
Oct 30 18:48:21.360453 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:21.359527    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:21.360688 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:21.360675    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:48:21.635201 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:21.630578    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 18:48:21.635201 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:21.630637    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" podUID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 18:48:21.729149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:21.728411    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:48:21.729149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:21.728506    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv" podUID="ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 18:48:22.421808 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:22.421263    4372 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:22.421808 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:22.421338    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" podUID="450e8016-b275-4172-ae20-5668c854e43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:22.631213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:22.630541    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 18:48:22.631213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:22.630597    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" podUID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 18:48:22.729234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:22.715290    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:48:22.729234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:22.715350    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv" podUID="ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 18:48:22.916471 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:22.916433    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:22.917169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:22.917108    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:22.929988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:22.929967    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 18:48:23.711991 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:23.711946    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:48:23.713372 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:23.713337    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv" podUID="ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 18:48:24.420004 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:24.419975    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:24.420817 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:24.420791    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:24.422275 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:24.422251    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 18:48:24.711672 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:24.711596    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:48:24.711816 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:24.711679    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv" podUID="ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 18:48:25.712395 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:25.712366    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:48:25.712783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:25.712411    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv" podUID="ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 18:48:26.711623 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:26.711586    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:48:26.711817 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:26.711643    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv" podUID="ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 18:48:27.547790 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.547756    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"]
Oct 30 18:48:27.551878 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.551863    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"
Oct 30 18:48:27.565110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.565091    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f4d94b\""
Oct 30 18:48:27.565517 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.565496    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f4d94b-dockercfg-tzgxd\""
Oct 30 18:48:27.570972 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.570948    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"]
Oct 30 18:48:27.575219 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.575199    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"
Oct 30 18:48:27.575323 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.575222    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"]
Oct 30 18:48:27.592745 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.592723    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"]
Oct 30 18:48:27.630515 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.630482    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a67bc25-5eff-4d3f-891b-f377ba0902f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4\" (UID: \"4a67bc25-5eff-4d3f-891b-f377ba0902f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"
Oct 30 18:48:27.630706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.630683    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gns9\" (UniqueName: \"kubernetes.io/projected/4a67bc25-5eff-4d3f-891b-f377ba0902f3-kube-api-access-8gns9\") pod \"payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4\" (UID: \"4a67bc25-5eff-4d3f-891b-f377ba0902f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"
Oct 30 18:48:27.718273 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.718239    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:48:27.718563 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.718532    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv" podUID="ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 18:48:27.751362 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.749115    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gns9\" (UniqueName: \"kubernetes.io/projected/4a67bc25-5eff-4d3f-891b-f377ba0902f3-kube-api-access-8gns9\") pod \"payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4\" (UID: \"4a67bc25-5eff-4d3f-891b-f377ba0902f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"
Oct 30 18:48:27.751362 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.749217    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smmvh\" (UniqueName: \"kubernetes.io/projected/5657423d-625a-4c36-9767-6427619d7a50-kube-api-access-smmvh\") pod \"payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl\" (UID: \"5657423d-625a-4c36-9767-6427619d7a50\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"
Oct 30 18:48:27.751362 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.749356    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a67bc25-5eff-4d3f-891b-f377ba0902f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4\" (UID: \"4a67bc25-5eff-4d3f-891b-f377ba0902f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"
Oct 30 18:48:27.751362 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.749394    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5657423d-625a-4c36-9767-6427619d7a50-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl\" (UID: \"5657423d-625a-4c36-9767-6427619d7a50\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"
Oct 30 18:48:27.751362 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.750977    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a67bc25-5eff-4d3f-891b-f377ba0902f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4\" (UID: \"4a67bc25-5eff-4d3f-891b-f377ba0902f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"
Oct 30 18:48:27.770850 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.770808    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gns9\" (UniqueName: \"kubernetes.io/projected/4a67bc25-5eff-4d3f-891b-f377ba0902f3-kube-api-access-8gns9\") pod \"payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4\" (UID: \"4a67bc25-5eff-4d3f-891b-f377ba0902f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"
Oct 30 18:48:27.854647 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.853000    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smmvh\" (UniqueName: \"kubernetes.io/projected/5657423d-625a-4c36-9767-6427619d7a50-kube-api-access-smmvh\") pod \"payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl\" (UID: \"5657423d-625a-4c36-9767-6427619d7a50\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"
Oct 30 18:48:27.854647 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.853173    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5657423d-625a-4c36-9767-6427619d7a50-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl\" (UID: \"5657423d-625a-4c36-9767-6427619d7a50\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"
Oct 30 18:48:27.854647 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.854041    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5657423d-625a-4c36-9767-6427619d7a50-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl\" (UID: \"5657423d-625a-4c36-9767-6427619d7a50\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"
Oct 30 18:48:27.872681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.872650    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"
Oct 30 18:48:27.886950 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.886910    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smmvh\" (UniqueName: \"kubernetes.io/projected/5657423d-625a-4c36-9767-6427619d7a50-kube-api-access-smmvh\") pod \"payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl\" (UID: \"5657423d-625a-4c36-9767-6427619d7a50\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"
Oct 30 18:48:27.921527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:27.920605    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"
Oct 30 18:48:27.933466 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:48:27.933429    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:48:28.208406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:28.208377    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"]
Oct 30 18:48:28.209458 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:48:28.209425    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5657423d_625a_4c36_9767_6427619d7a50.slice/crio-3c5dfe148fe8da76754266df63ad502b718cea9f10848c6edf52f1b90576d0d9 WatchSource:0}: Error finding container 3c5dfe148fe8da76754266df63ad502b718cea9f10848c6edf52f1b90576d0d9: Status 404 returned error can't find the container with id 3c5dfe148fe8da76754266df63ad502b718cea9f10848c6edf52f1b90576d0d9
Oct 30 18:48:28.217013 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:48:28.216989    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a67bc25_5eff_4d3f_891b_f377ba0902f3.slice/crio-92a4193624d24aa017aae3ea5b28e7c95404538fdfc552354c8faa43c2cf53e5 WatchSource:0}: Error finding container 92a4193624d24aa017aae3ea5b28e7c95404538fdfc552354c8faa43c2cf53e5: Status 404 returned error can't find the container with id 92a4193624d24aa017aae3ea5b28e7c95404538fdfc552354c8faa43c2cf53e5
Oct 30 18:48:28.218562 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:28.218445    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"]
Oct 30 18:48:28.484171 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:28.484036    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl" event={"ID":"5657423d-625a-4c36-9767-6427619d7a50","Type":"ContainerStarted","Data":"67b19d9f93636845ce9ccb77232b2e04b3107b4675074ab8e2b44df6dc5151ca"}
Oct 30 18:48:28.484171 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:28.484103    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"
Oct 30 18:48:28.484171 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:28.484141    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl" event={"ID":"5657423d-625a-4c36-9767-6427619d7a50","Type":"ContainerStarted","Data":"3c5dfe148fe8da76754266df63ad502b718cea9f10848c6edf52f1b90576d0d9"}
Oct 30 18:48:28.485939 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:28.485909    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4" event={"ID":"4a67bc25-5eff-4d3f-891b-f377ba0902f3","Type":"ContainerStarted","Data":"cf1cc81f5fa978c9074cb3d2d9ce98b767fc98ba73c2699563bd48a25d3d5b3f"}
Oct 30 18:48:28.486065 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:28.485964    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4" event={"ID":"4a67bc25-5eff-4d3f-891b-f377ba0902f3","Type":"ContainerStarted","Data":"92a4193624d24aa017aae3ea5b28e7c95404538fdfc552354c8faa43c2cf53e5"}
Oct 30 18:48:28.486065 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:28.486007    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"
Oct 30 18:48:28.507076 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:28.507030    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl" podStartSLOduration=1.507017161 podStartE2EDuration="1.507017161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:28.503411866 +0000 UTC m=+922.371830197" watchObservedRunningTime="2026-10-30 18:48:28.507017161 +0000 UTC m=+922.375435490"
Oct 30 18:48:28.524754 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:28.524717    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4" podStartSLOduration=1.524704324 podStartE2EDuration="1.524704324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:28.522969001 +0000 UTC m=+922.391387344" watchObservedRunningTime="2026-10-30 18:48:28.524704324 +0000 UTC m=+922.393122656"
Oct 30 18:48:28.940852 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:28.940824    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:28.941592 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:28.941155    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:28.941926 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:28.941907    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:48:32.398071 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.398035    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"]
Oct 30 18:48:32.406645 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.406603    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.423593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.423568    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.423718 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.423631    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.423846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.423823    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.423916 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.423860    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ssl\" (UniqueName: \"kubernetes.io/projected/8795b2cd-8ffe-4582-904d-2ae5751a2e09-kube-api-access-58ssl\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.423916 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.423889    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.424044 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.423944    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.424108 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.424051    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.424108 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.424088    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.424235 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.424162    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.441001 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.440973    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"]
Oct 30 18:48:32.525172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.525143    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.525327 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.525266    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.525327 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.525288    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ssl\" (UniqueName: \"kubernetes.io/projected/8795b2cd-8ffe-4582-904d-2ae5751a2e09-kube-api-access-58ssl\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.525327 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.525306    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.525327 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.525325    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.525554 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.525404    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.525554 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.525438    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.525554 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.525477    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.525732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.525576    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.525732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.525627    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.526002 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.525966    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.526002 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.525994    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.526210 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.526027    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.526278 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.526249    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.528060 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.528040    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.528203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.528144    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.540263 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.540241    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.540358 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.540314    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ssl\" (UniqueName: \"kubernetes.io/projected/8795b2cd-8ffe-4582-904d-2ae5751a2e09-kube-api-access-58ssl\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:32.725614 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:32.725553    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:33.014210 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:48:33.014180    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8795b2cd_8ffe_4582_904d_2ae5751a2e09.slice/crio-6f7953e6387e7e2f2d20abda44d91d44d0b1337b8c4745902145665305e4a764 WatchSource:0}: Error finding container 6f7953e6387e7e2f2d20abda44d91d44d0b1337b8c4745902145665305e4a764: Status 404 returned error can't find the container with id 6f7953e6387e7e2f2d20abda44d91d44d0b1337b8c4745902145665305e4a764
Oct 30 18:48:33.016322 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:33.015697    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"]
Oct 30 18:48:33.017723 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:33.017679    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:33.017826 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:33.017777    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:33.017893 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:33.017825    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:33.519341 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:33.519301    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" event={"ID":"8795b2cd-8ffe-4582-904d-2ae5751a2e09","Type":"ContainerStarted","Data":"4b65d792d8ac6aad87fc4063ff44ba654a52752f20760c1c5530b7b4176fe912"}
Oct 30 18:48:33.519888 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:33.519368    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" event={"ID":"8795b2cd-8ffe-4582-904d-2ae5751a2e09","Type":"ContainerStarted","Data":"6f7953e6387e7e2f2d20abda44d91d44d0b1337b8c4745902145665305e4a764"}
Oct 30 18:48:33.604415 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:33.604361    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" podStartSLOduration=1.604343297 podStartE2EDuration="1.604343297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:33.602532713 +0000 UTC m=+927.470951043" watchObservedRunningTime="2026-10-30 18:48:33.604343297 +0000 UTC m=+927.472761631"
Oct 30 18:48:33.726653 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:33.726577    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:34.366516 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.366482    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"]
Oct 30 18:48:34.370783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.370760    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:34.377748 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.376226    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f4d94b-serving-cert\""
Oct 30 18:48:34.415907 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.415870    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"]
Oct 30 18:48:34.473339 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.473307    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b5c108-eede-4a72-8335-2751062c90ec-maas-api-tls\") pod \"maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:34.473476 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.473360    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b5c108-eede-4a72-8335-2751062c90ec-openshift-service-ca\") pod \"maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:34.473476 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.473430    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc8hx\" (UniqueName: \"kubernetes.io/projected/f4b5c108-eede-4a72-8335-2751062c90ec-kube-api-access-kc8hx\") pod \"maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:34.473600 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.473550    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b5c108-eede-4a72-8335-2751062c90ec-metrics-certs\") pod \"maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:34.597151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.585211    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b5c108-eede-4a72-8335-2751062c90ec-maas-api-tls\") pod \"maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:34.597151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.585304    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b5c108-eede-4a72-8335-2751062c90ec-openshift-service-ca\") pod \"maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:34.597151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.585358    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc8hx\" (UniqueName: \"kubernetes.io/projected/f4b5c108-eede-4a72-8335-2751062c90ec-kube-api-access-kc8hx\") pod \"maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:34.597151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.585447    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b5c108-eede-4a72-8335-2751062c90ec-metrics-certs\") pod \"maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:34.597151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.592601    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b5c108-eede-4a72-8335-2751062c90ec-openshift-service-ca\") pod \"maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:34.612851 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.606886    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b5c108-eede-4a72-8335-2751062c90ec-metrics-certs\") pod \"maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:34.628174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.628008    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc8hx\" (UniqueName: \"kubernetes.io/projected/f4b5c108-eede-4a72-8335-2751062c90ec-kube-api-access-kc8hx\") pod \"maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:34.632765 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.632731    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b5c108-eede-4a72-8335-2751062c90ec-maas-api-tls\") pod \"maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:34.728470 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.728428    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:34.733259 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.733227    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:48:34.733489 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:34.733465    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:48:35.368326 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:35.368236    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"]
Oct 30 18:48:35.369240 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:35.369214    4372 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:48:35.588149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:35.585360    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp" event={"ID":"f4b5c108-eede-4a72-8335-2751062c90ec","Type":"ContainerStarted","Data":"a03eb6ef001f3afb80f377aa8525ed91c3f778bf11e8e78766936a996e6e4786"}
Oct 30 18:48:35.726356 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:35.726279    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:48:35.726356 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:35.726340    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:48:36.611595 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:36.611554    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp" event={"ID":"f4b5c108-eede-4a72-8335-2751062c90ec","Type":"ContainerStarted","Data":"793782c71835ebbd0645c9ca9d6c7cf2071a97220bfc709b8fc7a225539b0226"}
Oct 30 18:48:36.611779 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:36.611638    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:36.633983 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:36.633929    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp" podStartSLOduration=2.126659165 podStartE2EDuration="2.633912065s" totalImagesPullingTime="507.2529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:36.632862193 +0000 UTC m=+930.501280525" watchObservedRunningTime="2026-10-30 18:48:36.633912065 +0000 UTC m=+930.502330396"
Oct 30 18:48:36.726243 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:36.726214    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:48:36.726392 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:36.726262    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:48:37.095261 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:37.068146    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:48:37.726361 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:37.726319    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:48:37.726536 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:37.726383    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:48:38.728038 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:38.727507    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:48:38.728038 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:38.727578    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:48:39.300273 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:39.300222    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:39.338444 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:39.332218    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:39.432519 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:39.432478    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 18:48:39.448257 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:39.448206    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:39.519152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:39.518342    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"
Oct 30 18:48:39.542148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:39.539342    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"
Oct 30 18:48:39.789153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:39.782500    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:48:39.789153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:39.782563    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:48:39.875151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:39.874215    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 18:48:40.347322 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:40.347272    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:40.377381 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:40.377272    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:40.401869 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:40.401358    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:40.440594 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:40.438746    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:40.460152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:40.450073    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:40.463073 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:40.463041    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:48:40.467253 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:40.467216    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:48:40.467462 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:40.467447    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:48:40.733205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:40.731252    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:48:40.733205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:40.731318    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:48:40.875824 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:40.875779    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:40.901174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:40.901145    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:48:41.388150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:41.387364    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:41.388150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:41.388028    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:48:41.397234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:41.395357    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:41.426153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:41.420576    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:41.426153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:41.420949    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:48:41.478297 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:41.478241    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:41.479798 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:41.479052    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:41.731402 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:41.731310    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:48:41.744451 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:41.744407    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:48:41.771260 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:41.761261    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:41.771260 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:41.765201    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw" podUID="d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:41.916669 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:41.915533    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 18:48:41.917729 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:41.917676    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:41.918013 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:41.917969    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q" podUID="a5fb1131-af1f-4ce3-8822-c305e2767c89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:42.389151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:42.388214    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 18:48:42.389151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:42.388625    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Oct 30 18:48:42.461747 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:42.421214    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:42.465152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:42.462051    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:42.482161 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:42.481203    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 18:48:42.487002 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:42.486233    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 18:48:42.664490 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:42.663273    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:48:42.729541 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:42.729500    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:48:42.729819 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:42.729790    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:48:42.927203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:42.924305    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 18:48:42.945556 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:42.945522    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:42.945838 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:42.945801    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t" podUID="5567ca68-ab6f-496b-8538-c6102a04364c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:43.392926 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:43.392894    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:48:43.432543 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:43.388623    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:43.514178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:43.456384    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:43.514516 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:43.514474    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:43.552875 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:43.514666    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:43.725425 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:43.725303    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:43.725714 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:43.725665    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" podUID="72834257-61fb-4596-8b82-5917990f7469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:43.729602 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:43.728248    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:48:43.729602 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:43.728293    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:48:43.929177 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:43.924799    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:44.014473 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.014365    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:48:44.439102 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.437208    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:44.648156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.564444    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"]
Oct 30 18:48:44.692849 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.658461    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 18:48:44.692849 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.658810    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 18:48:44.772178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.734204    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:48:44.772178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.734264    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:48:44.851565 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.832666    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:44.888166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.887464    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f4b265-openshift-default-dockercfg-m6ntf\""
Oct 30 18:48:44.965689 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.938160    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-envoy\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:44.965689 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.938318    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-workload-certs\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:44.965689 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.938385    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-workload-socket\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:44.965689 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.938542    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-credential-socket\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:44.965689 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.938571    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-podinfo\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:44.965689 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.938669    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wrws\" (UniqueName: \"kubernetes.io/projected/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-kube-api-access-2wrws\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:44.965689 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.938705    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-data\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:44.965689 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.938732    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istiod-ca-cert\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:44.965689 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.938760    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-token\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:44.965689 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.938992    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 18:48:45.060439 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.060393    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:45.060714 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.060685    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv" podUID="ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:45.061142 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:44.900515    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"]
Oct 30 18:48:45.080149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.064665    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-workload-certs\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.080149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.064745    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-workload-socket\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.080149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.064886    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-credential-socket\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.080149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.064919    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-podinfo\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.080149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.065025    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wrws\" (UniqueName: \"kubernetes.io/projected/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-kube-api-access-2wrws\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.080149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.065064    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-data\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.080149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.065094    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istiod-ca-cert\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.080149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.065184    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-token\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.080149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.065224    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-envoy\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.081608 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.081148    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-workload-socket\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.081608 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.081428    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-workload-certs\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.083116 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.083072    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-credential-socket\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.084082 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.084063    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istiod-ca-cert\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.084551 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.084534    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-data\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.125712 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.125417    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-envoy\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.149764 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.149694    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-podinfo\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.256251 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.196938    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wrws\" (UniqueName: \"kubernetes.io/projected/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-kube-api-access-2wrws\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.285027 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.284988    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-token\") pod \"e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") " pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.386478 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.386423    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:45.464189 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.457201    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:45.535442 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.535334    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:45.669304 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.669265    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 18:48:45.669796 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.669767    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 18:48:45.753174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.745455    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:48:45.753174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.745510    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:48:45.853789 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:45.853647    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:46.239218 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:46.238823    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 18:48:46.479149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:46.474477    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 18:48:46.479149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:46.477870    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:46.557775 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:46.539216    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:46.557775 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:46.539562    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:48:46.695192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:46.680208    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 18:48:46.733163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:46.726564    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 18:48:46.740156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:46.734271    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:48:46.740431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:46.740396    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:48:47.005380 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:46.975705    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:48:47.042838 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.042796    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:48:47.048175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.048147    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:48:47.084958 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.084606    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:48:47.134863 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.134833    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:48:47.205270 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:48:47.205240    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9ec230_0cb3_4296_a9f1_58d09a8e17b4.slice/crio-002a4f733558b9dbe4554edcf3145da6cd334bbd1e5266de99fbf596290029b3 WatchSource:0}: Error finding container 002a4f733558b9dbe4554edcf3145da6cd334bbd1e5266de99fbf596290029b3: Status 404 returned error can't find the container with id 002a4f733558b9dbe4554edcf3145da6cd334bbd1e5266de99fbf596290029b3
Oct 30 18:48:47.206363 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.206335    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"]
Oct 30 18:48:47.207521 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.207482    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:47.207632 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.207564    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:47.207632 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.207605    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:47.520479 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.520374    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"]
Oct 30 18:48:47.525445 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.525416    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.563042 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.561978    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.563042 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.562057    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.563042 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.562084    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.563042 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.562185    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.563042 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.562213    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.563042 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.562321    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.563042 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.562388    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.563042 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.562413    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvjm6\" (UniqueName: \"kubernetes.io/projected/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-kube-api-access-vvjm6\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.563042 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.562449    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.563810 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.563788    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"]
Oct 30 18:48:47.687711 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.686365    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.687711 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.686442    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.687711 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.686469    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.687711 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.686540    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.687711 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.686566    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.687711 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.686682    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.687711 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.686750    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.687711 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.686774    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvjm6\" (UniqueName: \"kubernetes.io/projected/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-kube-api-access-vvjm6\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.687711 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.686810    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.702745 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.701682    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.713326 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.713094    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.713326 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.713276    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.713748 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.713702    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.713846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.713746    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.714262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.714237    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.777371 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.757532    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:47.777371 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.774867    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.824704 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.824666    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvjm6\" (UniqueName: \"kubernetes.io/projected/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-kube-api-access-vvjm6\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.826871 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.826723    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.830191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.830160    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9" event={"ID":"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4","Type":"ContainerStarted","Data":"317dd8af0e349a3c71a4f6aabfd87950085b9649de056d3d0447170817451d50"}
Oct 30 18:48:47.830316 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.830226    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9" event={"ID":"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4","Type":"ContainerStarted","Data":"002a4f733558b9dbe4554edcf3145da6cd334bbd1e5266de99fbf596290029b3"}
Oct 30 18:48:47.866220 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.866194    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:47.872368 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.872346    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:47.879418 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:47.879370    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9" podStartSLOduration=3.879354341 podStartE2EDuration="3.879354341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:47.875214379 +0000 UTC m=+941.743632710" watchObservedRunningTime="2026-10-30 18:48:47.879354341 +0000 UTC m=+941.747772675"
Oct 30 18:48:48.157617 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:48:48.157589    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18afbe35_49b3_4670_b3a1_ac9a184eeb8d.slice/crio-d36954780969e1e6d64ec6fa2381671234e695512120f07165208a5989b97b05 WatchSource:0}: Error finding container d36954780969e1e6d64ec6fa2381671234e695512120f07165208a5989b97b05: Status 404 returned error can't find the container with id d36954780969e1e6d64ec6fa2381671234e695512120f07165208a5989b97b05
Oct 30 18:48:48.157894 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:48.157872    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"]
Oct 30 18:48:48.160372 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:48.160338    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:48.160482 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:48.160405    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:48.160482 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:48.160438    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:48:48.386948 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:48.386910    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:48.394070 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:48.394041    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:48.859594 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:48.859550    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr" event={"ID":"18afbe35-49b3-4670-b3a1-ac9a184eeb8d","Type":"ContainerStarted","Data":"ad1f3f0ff0e4de03fa05edc0560c665b903c810a1b035a805260454a423e4933"}
Oct 30 18:48:48.860216 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:48.859619    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr" event={"ID":"18afbe35-49b3-4670-b3a1-ac9a184eeb8d","Type":"ContainerStarted","Data":"d36954780969e1e6d64ec6fa2381671234e695512120f07165208a5989b97b05"}
Oct 30 18:48:48.871149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:48.862255    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:48.871149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:48.862407    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:48:48.871599 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:48.871576    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:48.937792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:48.937736    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr" podStartSLOduration=1.9377182080000002 podStartE2EDuration="1.937718208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:48.93697165 +0000 UTC m=+942.805389984" watchObservedRunningTime="2026-10-30 18:48:48.937718208 +0000 UTC m=+942.806136538"
Oct 30 18:48:49.492151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:49.487463    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 18:48:49.868406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:49.868374    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 18:48:49.868811 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:49.868426    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr" podUID="18afbe35-49b3-4670-b3a1-ac9a184eeb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 18:48:50.867296 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:50.867262    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 18:48:50.867488 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:50.867317    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr" podUID="18afbe35-49b3-4670-b3a1-ac9a184eeb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 18:48:51.866516 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:51.866487    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 18:48:51.866941 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:51.866534    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr" podUID="18afbe35-49b3-4670-b3a1-ac9a184eeb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 18:48:52.311164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:52.311045    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"]
Oct 30 18:48:52.315396 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:52.315369    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"
Oct 30 18:48:52.325806 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:52.325779    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qj6b\" (UniqueName: \"kubernetes.io/projected/02abb3d7-836b-46e7-b2d1-b130e2a60bfe-kube-api-access-9qj6b\") pod \"payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn\" (UID: \"02abb3d7-836b-46e7-b2d1-b130e2a60bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"
Oct 30 18:48:52.326045 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:52.326022    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02abb3d7-836b-46e7-b2d1-b130e2a60bfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn\" (UID: \"02abb3d7-836b-46e7-b2d1-b130e2a60bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"
Oct 30 18:48:52.357239 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:52.357213    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"]
Oct 30 18:48:52.361586 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:52.361557    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f4b265\""
Oct 30 18:48:52.428253 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:52.428219    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qj6b\" (UniqueName: \"kubernetes.io/projected/02abb3d7-836b-46e7-b2d1-b130e2a60bfe-kube-api-access-9qj6b\") pod \"payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn\" (UID: \"02abb3d7-836b-46e7-b2d1-b130e2a60bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"
Oct 30 18:48:52.428448 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:52.428414    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02abb3d7-836b-46e7-b2d1-b130e2a60bfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn\" (UID: \"02abb3d7-836b-46e7-b2d1-b130e2a60bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"
Oct 30 18:48:52.429449 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:52.429419    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02abb3d7-836b-46e7-b2d1-b130e2a60bfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn\" (UID: \"02abb3d7-836b-46e7-b2d1-b130e2a60bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"
Oct 30 18:48:52.447202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:52.447176    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qj6b\" (UniqueName: \"kubernetes.io/projected/02abb3d7-836b-46e7-b2d1-b130e2a60bfe-kube-api-access-9qj6b\") pod \"payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn\" (UID: \"02abb3d7-836b-46e7-b2d1-b130e2a60bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"
Oct 30 18:48:52.637159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:52.637062    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"
Oct 30 18:48:52.867167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:52.867110    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 18:48:52.867625 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:52.867209    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr" podUID="18afbe35-49b3-4670-b3a1-ac9a184eeb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 18:48:53.131612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:53.131582    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"]
Oct 30 18:48:53.131988 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:48:53.131965    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02abb3d7_836b_46e7_b2d1_b130e2a60bfe.slice/crio-1bb4c9e29872d6979556572a629b03dc2e0e57062572cc544c67165e76edd72b WatchSource:0}: Error finding container 1bb4c9e29872d6979556572a629b03dc2e0e57062572cc544c67165e76edd72b: Status 404 returned error can't find the container with id 1bb4c9e29872d6979556572a629b03dc2e0e57062572cc544c67165e76edd72b
Oct 30 18:48:53.872168 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:53.871209    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 18:48:53.872168 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:53.871276    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr" podUID="18afbe35-49b3-4670-b3a1-ac9a184eeb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 18:48:53.935973 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:53.935932    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn" event={"ID":"02abb3d7-836b-46e7-b2d1-b130e2a60bfe","Type":"ContainerStarted","Data":"141c44acdc6b6ec19386c8baaee25b7ed284377137da40e44484ef748e6a4575"}
Oct 30 18:48:53.936164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:53.935991    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn" event={"ID":"02abb3d7-836b-46e7-b2d1-b130e2a60bfe","Type":"ContainerStarted","Data":"1bb4c9e29872d6979556572a629b03dc2e0e57062572cc544c67165e76edd72b"}
Oct 30 18:48:53.936164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:53.936012    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"
Oct 30 18:48:54.003363 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:54.003296    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn" podStartSLOduration=2.003276901 podStartE2EDuration="2.003276901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:54.001403147 +0000 UTC m=+947.869821490" watchObservedRunningTime="2026-10-30 18:48:54.003276901 +0000 UTC m=+947.871695238"
Oct 30 18:48:54.867393 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:54.867360    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 18:48:54.867563 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:54.867415    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr" podUID="18afbe35-49b3-4670-b3a1-ac9a184eeb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 18:48:55.872687 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:55.872656    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:55.873141 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:55.873108    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:55.873906 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:48:55.873883    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:48:57.697044 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:48:57.697010    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:49:02.627495 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.627464    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"]
Oct 30 18:49:02.634709 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.634685    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:02.641912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.641887    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"]
Oct 30 18:49:02.648935 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.648336    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f4b265-serving-cert\""
Oct 30 18:49:02.682179 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.682111    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2lg7\" (UniqueName: \"kubernetes.io/projected/8745c459-008e-471d-a4b4-67da58c8432d-kube-api-access-z2lg7\") pod \"maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:02.682321 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.682226    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8745c459-008e-471d-a4b4-67da58c8432d-maas-api-tls\") pod \"maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:02.682390 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.682331    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8745c459-008e-471d-a4b4-67da58c8432d-metrics-certs\") pod \"maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:02.682614 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.682589    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8745c459-008e-471d-a4b4-67da58c8432d-openshift-service-ca\") pod \"maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:02.795165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.785159    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2lg7\" (UniqueName: \"kubernetes.io/projected/8745c459-008e-471d-a4b4-67da58c8432d-kube-api-access-z2lg7\") pod \"maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:02.795165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.785269    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8745c459-008e-471d-a4b4-67da58c8432d-maas-api-tls\") pod \"maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:02.795165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.785528    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8745c459-008e-471d-a4b4-67da58c8432d-metrics-certs\") pod \"maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:02.795165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.785699    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8745c459-008e-471d-a4b4-67da58c8432d-openshift-service-ca\") pod \"maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:02.795165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.786582    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8745c459-008e-471d-a4b4-67da58c8432d-openshift-service-ca\") pod \"maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:02.869217 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.852362    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2lg7\" (UniqueName: \"kubernetes.io/projected/8745c459-008e-471d-a4b4-67da58c8432d-kube-api-access-z2lg7\") pod \"maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:02.885291 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.884034    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8745c459-008e-471d-a4b4-67da58c8432d-maas-api-tls\") pod \"maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:02.898433 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.890683    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8745c459-008e-471d-a4b4-67da58c8432d-metrics-certs\") pod \"maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:02.955620 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:02.955586    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:03.216561 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:03.216483    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"]
Oct 30 18:49:03.222446 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:03.222420    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"
Oct 30 18:49:03.243589 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:03.241072    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"]
Oct 30 18:49:03.244681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:03.244653    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f44051\""
Oct 30 18:49:03.293472 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:03.293443    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"]
Oct 30 18:49:03.294967 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:49:03.294941    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8745c459_008e_471d_a4b4_67da58c8432d.slice/crio-1bdda4d9cb4ed6b2c610885da341ca5346c3719fc7d63ee194f0d7de297a1fdb WatchSource:0}: Error finding container 1bdda4d9cb4ed6b2c610885da341ca5346c3719fc7d63ee194f0d7de297a1fdb: Status 404 returned error can't find the container with id 1bdda4d9cb4ed6b2c610885da341ca5346c3719fc7d63ee194f0d7de297a1fdb
Oct 30 18:49:03.308400 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:03.308374    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01404f15-9f9f-446c-88d0-83e2f67bd877-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk\" (UID: \"01404f15-9f9f-446c-88d0-83e2f67bd877\") " pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"
Oct 30 18:49:03.308534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:03.308516    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqbbd\" (UniqueName: \"kubernetes.io/projected/01404f15-9f9f-446c-88d0-83e2f67bd877-kube-api-access-tqbbd\") pod \"payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk\" (UID: \"01404f15-9f9f-446c-88d0-83e2f67bd877\") " pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"
Oct 30 18:49:03.507200 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:03.443497    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqbbd\" (UniqueName: \"kubernetes.io/projected/01404f15-9f9f-446c-88d0-83e2f67bd877-kube-api-access-tqbbd\") pod \"payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk\" (UID: \"01404f15-9f9f-446c-88d0-83e2f67bd877\") " pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"
Oct 30 18:49:03.507200 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:03.443765    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01404f15-9f9f-446c-88d0-83e2f67bd877-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk\" (UID: \"01404f15-9f9f-446c-88d0-83e2f67bd877\") " pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"
Oct 30 18:49:03.507200 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:03.444753    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01404f15-9f9f-446c-88d0-83e2f67bd877-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk\" (UID: \"01404f15-9f9f-446c-88d0-83e2f67bd877\") " pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"
Oct 30 18:49:03.613154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:03.584892    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqbbd\" (UniqueName: \"kubernetes.io/projected/01404f15-9f9f-446c-88d0-83e2f67bd877-kube-api-access-tqbbd\") pod \"payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk\" (UID: \"01404f15-9f9f-446c-88d0-83e2f67bd877\") " pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"
Oct 30 18:49:03.623073 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:03.623042    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"
Oct 30 18:49:04.093036 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:04.093006    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"]
Oct 30 18:49:04.093501 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:49:04.093241    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01404f15_9f9f_446c_88d0_83e2f67bd877.slice/crio-4af442e5388a21f04c72bc5a562decd841f03aa4be3b83e41a216a74f7cfe17f WatchSource:0}: Error finding container 4af442e5388a21f04c72bc5a562decd841f03aa4be3b83e41a216a74f7cfe17f: Status 404 returned error can't find the container with id 4af442e5388a21f04c72bc5a562decd841f03aa4be3b83e41a216a74f7cfe17f
Oct 30 18:49:04.096053 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:04.096023    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb" event={"ID":"8745c459-008e-471d-a4b4-67da58c8432d","Type":"ContainerStarted","Data":"5103b96bd8a32cdebe35fc60691f9d9adb7685067c32e8e71fc5f37f3bf28a99"}
Oct 30 18:49:04.096217 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:04.096081    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb" event={"ID":"8745c459-008e-471d-a4b4-67da58c8432d","Type":"ContainerStarted","Data":"1bdda4d9cb4ed6b2c610885da341ca5346c3719fc7d63ee194f0d7de297a1fdb"}
Oct 30 18:49:04.096345 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:04.096299    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:04.121639 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:04.121567    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb" podStartSLOduration=1.544632269 podStartE2EDuration="2.121553566s" totalImagesPullingTime="576.921297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:04.119642066 +0000 UTC m=+957.988060408" watchObservedRunningTime="2026-10-30 18:49:04.121553566 +0000 UTC m=+957.989971896"
Oct 30 18:49:04.973860 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:04.973826    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"
Oct 30 18:49:05.200252 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:05.200213    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk" event={"ID":"01404f15-9f9f-446c-88d0-83e2f67bd877","Type":"ContainerStarted","Data":"a5c86532cadab9838b4d4da0117400fabeec91a3ebf1888518e9aefc84ec1149"}
Oct 30 18:49:05.200862 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:05.200833    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk" event={"ID":"01404f15-9f9f-446c-88d0-83e2f67bd877","Type":"ContainerStarted","Data":"4af442e5388a21f04c72bc5a562decd841f03aa4be3b83e41a216a74f7cfe17f"}
Oct 30 18:49:05.202673 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:05.202652    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"
Oct 30 18:49:06.401153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:06.399157    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:06.452156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:06.449912    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 18:49:06.483664 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:06.483617    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:06.487236 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:06.487202    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:07.123379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:06.921925    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:07.123379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:07.064241    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:07.400207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:07.399506    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:07.400207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:07.399861    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:49:07.427422 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:07.427387    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:07.442683 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:07.442611    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk" podStartSLOduration=4.442591237 podStartE2EDuration="4.442591237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:05.269490863 +0000 UTC m=+959.137909196" watchObservedRunningTime="2026-10-30 18:49:07.442591237 +0000 UTC m=+961.311009567"
Oct 30 18:49:07.579617 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:07.579560    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 18:49:07.602060 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:07.602024    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:49:07.690212 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:07.654267    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:49:07.690212 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:07.655302    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:49:07.948880 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:07.948792    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded"
Oct 30 18:49:07.950257 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:07.949859    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:49:07.962024 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:07.961997    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:49:07.962594 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:07.962579    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:49:10.421446 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:10.421412    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:49:17.085735 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.085460    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"]
Oct 30 18:49:17.090869 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.090841    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.107934 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.107905    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-30970e-openshift-default-dockercfg-4tktt\""
Oct 30 18:49:17.111796 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.111771    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"]
Oct 30 18:49:17.177587 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.177556    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-podinfo\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.177738 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.177665    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-workload-certs\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.177738 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.177721    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmnhf\" (UniqueName: \"kubernetes.io/projected/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-kube-api-access-qmnhf\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.177848 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.177779    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-token\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.177898 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.177838    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istiod-ca-cert\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.177987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.177966    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-envoy\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.178054 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.178038    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-credential-socket\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.178097 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.178088    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-workload-socket\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.178193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.178179    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-data\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.293155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.281067    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-token\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.293155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.281146    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istiod-ca-cert\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.293155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.281220    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-envoy\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.293155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.281315    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-credential-socket\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.293155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.281374    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-workload-socket\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.293155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.281453    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-data\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.293155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.281507    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-podinfo\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.293155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.281583    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-workload-certs\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.293155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.281651    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnhf\" (UniqueName: \"kubernetes.io/projected/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-kube-api-access-qmnhf\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.293155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.283053    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istiod-ca-cert\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.307154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.305607    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-data\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.307154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.305875    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-credential-socket\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.307154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.306104    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-workload-socket\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.315150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.312515    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-workload-certs\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.315150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.312776    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-envoy\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.329148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.328017    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-podinfo\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.374582 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.353600    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"
Oct 30 18:49:17.374582 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.364647    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-token\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.376521 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.376488    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnhf\" (UniqueName: \"kubernetes.io/projected/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-kube-api-access-qmnhf\") pod \"e2e-api-b-30970e-openshift-default-d96c4969f-drgkn\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") " pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.430150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.419811    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:17.566917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.566883    4372 generic.go:356] "Generic (PLEG): container finished" podID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerID="903c5942727be773f1336a3b5866d3e00b05992470e4bc2466b3f43868ef57af" exitCode=0
Oct 30 18:49:17.567253 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.567044    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf" event={"ID":"caf281ca-692b-48c8-84d7-13ca8e6e7489","Type":"ContainerDied","Data":"903c5942727be773f1336a3b5866d3e00b05992470e4bc2466b3f43868ef57af"}
Oct 30 18:49:17.567494 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.567468    4372 scope.go:122] "RemoveContainer" containerID="903c5942727be773f1336a3b5866d3e00b05992470e4bc2466b3f43868ef57af"
Oct 30 18:49:17.569827 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.569347    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:49:17.569827 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.569417    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:49:17.569827 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.569450    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:49:17.890229 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.889494    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"]
Oct 30 18:49:17.909803 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.909742    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:49:17.910084 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.910051    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:49:17.910471 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:17.910445    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:49:18.575790 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:18.575748    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn" event={"ID":"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3","Type":"ContainerStarted","Data":"665eeabdf2bc826c6ee54928b394063be812396e1cc4002b419620c08d44fb39"}
Oct 30 18:49:18.576299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:18.575813    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn" event={"ID":"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3","Type":"ContainerStarted","Data":"1ff303353df1ba60f3693986102267d088c7e8a73a4f149ad12872ded0750b23"}
Oct 30 18:49:18.577866 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:18.577841    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf" event={"ID":"caf281ca-692b-48c8-84d7-13ca8e6e7489","Type":"ContainerStarted","Data":"6526dc920598f47b104a5557e7057e2e2d76dfa471adc8f8057a460484a387cb"}
Oct 30 18:49:18.611731 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:18.611676    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn" podStartSLOduration=1.611660755 podStartE2EDuration="1.611660755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:18.606940082 +0000 UTC m=+972.475358413" watchObservedRunningTime="2026-10-30 18:49:18.611660755 +0000 UTC m=+972.480079084"
Oct 30 18:49:18.795823 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:18.795736    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:49:18.796022 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:18.795826    4372 patch_prober.go:29] interesting pod/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:49:18.796022 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:18.795877    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf" podUID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 18:49:19.421416 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:19.421382    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:19.426792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:19.426767    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:19.636248 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:19.636214    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:19.652188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:19.648512    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:49:19.824676 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:19.824531    4372 patch_prober.go:29] interesting pod/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:49:19.824676 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:19.824604    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf" podUID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 18:49:20.816150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:20.805104    4372 patch_prober.go:29] interesting pod/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:49:20.816150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:20.805202    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf" podUID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 18:49:21.360425 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:21.360393    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:49:21.668807 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:21.668729    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"]
Oct 30 18:49:21.673399 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:21.673383    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"
Oct 30 18:49:21.707238 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:21.706878    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f9521e-dockercfg-ls24r\""
Oct 30 18:49:21.719623 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:21.719598    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"]
Oct 30 18:49:21.811498 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:21.811467    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f9521e\""
Oct 30 18:49:21.817223 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:21.812920    4372 patch_prober.go:29] interesting pod/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:49:21.818701 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:21.818339    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f48tt\" (UniqueName: \"kubernetes.io/projected/af1375ab-417e-4a88-880e-f3f7a171d63e-kube-api-access-f48tt\") pod \"payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb\" (UID: \"af1375ab-417e-4a88-880e-f3f7a171d63e\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"
Oct 30 18:49:21.818701 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:21.818557    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1375ab-417e-4a88-880e-f3f7a171d63e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb\" (UID: \"af1375ab-417e-4a88-880e-f3f7a171d63e\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"
Oct 30 18:49:21.818953 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:21.818083    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf" podUID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 18:49:21.925615 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:21.925525    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1375ab-417e-4a88-880e-f3f7a171d63e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb\" (UID: \"af1375ab-417e-4a88-880e-f3f7a171d63e\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"
Oct 30 18:49:21.927072 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:21.927044    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f48tt\" (UniqueName: \"kubernetes.io/projected/af1375ab-417e-4a88-880e-f3f7a171d63e-kube-api-access-f48tt\") pod \"payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb\" (UID: \"af1375ab-417e-4a88-880e-f3f7a171d63e\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"
Oct 30 18:49:21.927661 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:21.926766    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1375ab-417e-4a88-880e-f3f7a171d63e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb\" (UID: \"af1375ab-417e-4a88-880e-f3f7a171d63e\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"
Oct 30 18:49:21.952938 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:21.952905    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f48tt\" (UniqueName: \"kubernetes.io/projected/af1375ab-417e-4a88-880e-f3f7a171d63e-kube-api-access-f48tt\") pod \"payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb\" (UID: \"af1375ab-417e-4a88-880e-f3f7a171d63e\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"
Oct 30 18:49:22.016657 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:22.016624    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"
Oct 30 18:49:22.775388 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:22.775360    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"]
Oct 30 18:49:22.775744 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:49:22.775716    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1375ab_417e_4a88_880e_f3f7a171d63e.slice/crio-f618fc760b7d35a27688f3d3f6fa61f8e12129f1461f027365d49f2be80b6988 WatchSource:0}: Error finding container f618fc760b7d35a27688f3d3f6fa61f8e12129f1461f027365d49f2be80b6988: Status 404 returned error can't find the container with id f618fc760b7d35a27688f3d3f6fa61f8e12129f1461f027365d49f2be80b6988
Oct 30 18:49:22.803110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:22.802697    4372 patch_prober.go:29] interesting pod/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:49:22.803110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:22.802749    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf" podUID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 18:49:23.655814 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:23.655775    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb" event={"ID":"af1375ab-417e-4a88-880e-f3f7a171d63e","Type":"ContainerStarted","Data":"56ff4b9ae548218030f58ce47b20fcb0e9e9fd2084a0fe81eb7f55ddeeaa5de3"}
Oct 30 18:49:23.656246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:23.655829    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb" event={"ID":"af1375ab-417e-4a88-880e-f3f7a171d63e","Type":"ContainerStarted","Data":"f618fc760b7d35a27688f3d3f6fa61f8e12129f1461f027365d49f2be80b6988"}
Oct 30 18:49:23.656246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:23.655858    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"
Oct 30 18:49:23.708153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:23.707834    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb" podStartSLOduration=2.707811482 podStartE2EDuration="2.707811482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:23.686513557 +0000 UTC m=+977.554931889" watchObservedRunningTime="2026-10-30 18:49:23.707811482 +0000 UTC m=+977.576229813"
Oct 30 18:49:23.803632 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:23.803259    4372 patch_prober.go:29] interesting pod/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:49:23.803632 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:23.803321    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf" podUID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 18:49:24.799718 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:24.799687    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:49:24.800834 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:24.800810    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:49:27.697662 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:49:27.697624    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:49:28.409240 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:28.409208    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:49:31.655527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.655492    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"]
Oct 30 18:49:31.663566 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.663543    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:31.671254 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.671225    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"]
Oct 30 18:49:31.673531 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.673463    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f9521e-serving-cert\""
Oct 30 18:49:31.771736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.771705    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-maas-api-tls\") pod \"maas-api-e2e-api-b-f9521e-df56d685-gbh6x\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:31.771882 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.771808    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-openshift-service-ca\") pod \"maas-api-e2e-api-b-f9521e-df56d685-gbh6x\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:31.771965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.771949    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt588\" (UniqueName: \"kubernetes.io/projected/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-kube-api-access-rt588\") pod \"maas-api-e2e-api-b-f9521e-df56d685-gbh6x\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:31.772024 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.772011    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-metrics-certs\") pod \"maas-api-e2e-api-b-f9521e-df56d685-gbh6x\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:31.884155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.874466    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-maas-api-tls\") pod \"maas-api-e2e-api-b-f9521e-df56d685-gbh6x\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:31.884155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.874604    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-openshift-service-ca\") pod \"maas-api-e2e-api-b-f9521e-df56d685-gbh6x\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:31.884155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.874840    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt588\" (UniqueName: \"kubernetes.io/projected/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-kube-api-access-rt588\") pod \"maas-api-e2e-api-b-f9521e-df56d685-gbh6x\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:31.884155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.874925    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-metrics-certs\") pod \"maas-api-e2e-api-b-f9521e-df56d685-gbh6x\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:31.893154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.891822    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-metrics-certs\") pod \"maas-api-e2e-api-b-f9521e-df56d685-gbh6x\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:31.899149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.895742    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-openshift-service-ca\") pod \"maas-api-e2e-api-b-f9521e-df56d685-gbh6x\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:31.925199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.915734    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-maas-api-tls\") pod \"maas-api-e2e-api-b-f9521e-df56d685-gbh6x\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:31.936147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:31.932708    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt588\" (UniqueName: \"kubernetes.io/projected/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-kube-api-access-rt588\") pod \"maas-api-e2e-api-b-f9521e-df56d685-gbh6x\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:32.015830 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:32.015793    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:32.388287 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:32.388227    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"]
Oct 30 18:49:32.772802 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:32.772708    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x" event={"ID":"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6","Type":"ContainerStarted","Data":"e855487da9cacd91e8569be8a3e3b9fe3674e6859db06000f169f7a05b7497e7"}
Oct 30 18:49:33.760266 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.760228    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"]
Oct 30 18:49:33.765833 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.765807    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:33.777010 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.776975    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-30970e-serving-cert\""
Oct 30 18:49:33.816286 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.816258    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"]
Oct 30 18:49:33.820753 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.820723    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x" event={"ID":"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6","Type":"ContainerStarted","Data":"9a09359aedff21a90a4ecc785ef70896a1c61abdff34409a546e38191494e5cb"}
Oct 30 18:49:33.821059 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.821041    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:33.825257 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.825232    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-maas-api-tls\") pod \"maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:33.825370 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.825285    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpvgb\" (UniqueName: \"kubernetes.io/projected/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-kube-api-access-fpvgb\") pod \"maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:33.825370 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.825340    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-metrics-certs\") pod \"maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:33.825492 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.825443    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-openshift-service-ca\") pod \"maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:33.843409 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.843362    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x" podStartSLOduration=2.231636063 podStartE2EDuration="2.843349551s" totalImagesPullingTime="611.713488ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:33.841454512 +0000 UTC m=+987.709872844" watchObservedRunningTime="2026-10-30 18:49:33.843349551 +0000 UTC m=+987.711767873"
Oct 30 18:49:33.926447 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.926410    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-openshift-service-ca\") pod \"maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:33.926656 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.926637    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-maas-api-tls\") pod \"maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:33.926725 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.926699    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpvgb\" (UniqueName: \"kubernetes.io/projected/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-kube-api-access-fpvgb\") pod \"maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:33.926791 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:49:33.926742    4372 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-serving-cert: secret "maas-api-e2e-api-b-30970e-serving-cert" not found
Oct 30 18:49:33.926791 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.926754    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-metrics-certs\") pod \"maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:33.926917 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:49:33.926821    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-maas-api-tls podName:2a668eb1-f2a4-4819-8d77-2b82b4884a8d nodeName:}" failed. No retries permitted until 2026-10-30 18:49:34.426790033 +0000 UTC m=+988.295208347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-maas-api-tls") pod "maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5" (UID: "2a668eb1-f2a4-4819-8d77-2b82b4884a8d") : secret "maas-api-e2e-api-b-30970e-serving-cert" not found
Oct 30 18:49:33.927250 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.927230    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-openshift-service-ca\") pod \"maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:33.929518 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.929491    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-metrics-certs\") pod \"maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:33.942069 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:33.942049    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpvgb\" (UniqueName: \"kubernetes.io/projected/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-kube-api-access-fpvgb\") pod \"maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:34.438691 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:34.438657    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-maas-api-tls\") pod \"maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:34.441224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:34.441204    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-maas-api-tls\") pod \"maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:34.676159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:34.669072    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"
Oct 30 18:49:34.723207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:34.720978    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:35.215554 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:49:35.215516    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a668eb1_f2a4_4819_8d77_2b82b4884a8d.slice/crio-3707ed6d8de14f30d4ff37b9ba3a1519aedcef234bc819979e1d508dc3a94553 WatchSource:0}: Error finding container 3707ed6d8de14f30d4ff37b9ba3a1519aedcef234bc819979e1d508dc3a94553: Status 404 returned error can't find the container with id 3707ed6d8de14f30d4ff37b9ba3a1519aedcef234bc819979e1d508dc3a94553
Oct 30 18:49:35.215896 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:35.215638    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"]
Oct 30 18:49:35.868384 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:35.868345    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5" event={"ID":"2a668eb1-f2a4-4819-8d77-2b82b4884a8d","Type":"ContainerStarted","Data":"7c474fec40d0084a7087ef5469f85c3c46ca671c40b01b43e58a3d46fac89a5b"}
Oct 30 18:49:35.868541 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:35.868417    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5" event={"ID":"2a668eb1-f2a4-4819-8d77-2b82b4884a8d","Type":"ContainerStarted","Data":"3707ed6d8de14f30d4ff37b9ba3a1519aedcef234bc819979e1d508dc3a94553"}
Oct 30 18:49:35.869501 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:35.869479    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:35.894325 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:35.894235    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5" podStartSLOduration=2.467927812 podStartE2EDuration="2.894217942s" totalImagesPullingTime="426.29013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:35.890829603 +0000 UTC m=+989.759247934" watchObservedRunningTime="2026-10-30 18:49:35.894217942 +0000 UTC m=+989.762636275"
Oct 30 18:49:39.084575 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:39.084547    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"]
Oct 30 18:49:39.099359 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:39.099318    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"]
Oct 30 18:49:39.878624 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:39.878579    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:49:40.922066 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:40.922035    4372 generic.go:356] "Generic (PLEG): container finished" podID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" containerID="317dd8af0e349a3c71a4f6aabfd87950085b9649de056d3d0447170817451d50" exitCode=0
Oct 30 18:49:40.922609 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:40.922111    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9" event={"ID":"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4","Type":"ContainerDied","Data":"317dd8af0e349a3c71a4f6aabfd87950085b9649de056d3d0447170817451d50"}
Oct 30 18:49:40.922609 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:40.922502    4372 scope.go:122] "RemoveContainer" containerID="317dd8af0e349a3c71a4f6aabfd87950085b9649de056d3d0447170817451d50"
Oct 30 18:49:40.924009 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:40.923982    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:49:40.924080 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:40.924052    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:49:40.924120 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:40.924084    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:49:41.932378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:41.932349    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9" event={"ID":"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4","Type":"ContainerStarted","Data":"c2bd126e2fec738bc5193b25692603ac8c9d03917373707935590e44f5c2743c"}
Oct 30 18:49:42.387547 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.387514    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:49:42.387908 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.387884    4372 patch_prober.go:29] interesting pod/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 18:49:42.388073 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.388047    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9" podUID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 18:49:42.909150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.905736    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"]
Oct 30 18:49:42.911226 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.911203    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"]
Oct 30 18:49:42.911370 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.911304    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:42.915335 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.915308    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:49:42.922034 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.921695    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f44051-openshift-default-dockercfg-5wcgd\""
Oct 30 18:49:42.940744 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.940720    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-workload-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:42.941262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.940843    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-workload-certs\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:42.941262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.941017    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-envoy\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:42.941262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.941200    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-credential-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:42.941425 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.941274    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdfd538-a850-48e9-883f-f7428503b8a9-istiod-ca-cert\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:42.941425 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.941341    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-data\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:42.941425 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.941407    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-podinfo\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:42.941537 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.941460    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-token\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:42.941537 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:42.941515    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd9dn\" (UniqueName: \"kubernetes.io/projected/dcdfd538-a850-48e9-883f-f7428503b8a9-kube-api-access-vd9dn\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.042790 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.042746    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-workload-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.042982 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.042811    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-workload-certs\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.042982 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.042887    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-envoy\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.043118 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.043001    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-credential-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.043118 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.043065    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdfd538-a850-48e9-883f-f7428503b8a9-istiod-ca-cert\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.043118 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.043089    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-workload-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.043118 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.043111    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-data\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.043366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.043205    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-podinfo\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.043366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.043234    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-token\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.043366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.043311    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd9dn\" (UniqueName: \"kubernetes.io/projected/dcdfd538-a850-48e9-883f-f7428503b8a9-kube-api-access-vd9dn\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.043920 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.043891    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdfd538-a850-48e9-883f-f7428503b8a9-istiod-ca-cert\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.044035 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.043989    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-data\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.044097 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.044064    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-workload-certs\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.044214 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.044197    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-credential-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.045966 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.045942    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-envoy\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.046676 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.046652    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-podinfo\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.057926 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.057901    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd9dn\" (UniqueName: \"kubernetes.io/projected/dcdfd538-a850-48e9-883f-f7428503b8a9-kube-api-access-vd9dn\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.060830 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.060252    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-token\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-67k69\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.260239 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.230439    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:43.400154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.390567    4372 patch_prober.go:29] interesting pod/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 18:49:43.400154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.390627    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9" podUID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 18:49:43.615364 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.615318    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"]
Oct 30 18:49:43.619193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.618719    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:49:43.619193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.618794    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:49:43.619193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:43.618845    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:49:44.005205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.004145    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69" event={"ID":"dcdfd538-a850-48e9-883f-f7428503b8a9","Type":"ContainerStarted","Data":"7451ab21d620e6bf75ec88c0997ab82aeaf30f64bea334f9e2a2380b8bd9a87c"}
Oct 30 18:49:44.005205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.004234    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69" event={"ID":"dcdfd538-a850-48e9-883f-f7428503b8a9","Type":"ContainerStarted","Data":"7d698369a2266df5ed0f1086cf45af38613376f391ec956fcbb2b2f5099c89a5"}
Oct 30 18:49:44.103151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.102634    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4" podUID="4a67bc25-5eff-4d3f-891b-f377ba0902f3" containerName="payload-pre-processing" containerID="cri-o://cf1cc81f5fa978c9074cb3d2d9ce98b767fc98ba73c2699563bd48a25d3d5b3f" gracePeriod=25
Oct 30 18:49:44.149745 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.122810    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl" podUID="5657423d-625a-4c36-9767-6427619d7a50" containerName="payload-processing" containerID="cri-o://67b19d9f93636845ce9ccb77232b2e04b3107b4675074ab8e2b44df6dc5151ca" gracePeriod=25
Oct 30 18:49:44.233203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.231863    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:44.235227 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.234885    4372 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 18:49:44.235227 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.234942    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69" podUID="dcdfd538-a850-48e9-883f-f7428503b8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 18:49:44.411665 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.390045    4372 patch_prober.go:29] interesting pod/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 18:49:44.411665 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.390106    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9" podUID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 18:49:44.613323 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.612968    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"
Oct 30 18:49:44.638455 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.638431    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"
Oct 30 18:49:44.647449 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.647423    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a67bc25-5eff-4d3f-891b-f377ba0902f3-kube-api-access-8gns9\" (UniqueName: \"kubernetes.io/projected/4a67bc25-5eff-4d3f-891b-f377ba0902f3-kube-api-access-8gns9\") pod \"4a67bc25-5eff-4d3f-891b-f377ba0902f3\" (UID: \"4a67bc25-5eff-4d3f-891b-f377ba0902f3\") "
Oct 30 18:49:44.647625 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.647570    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a67bc25-5eff-4d3f-891b-f377ba0902f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a67bc25-5eff-4d3f-891b-f377ba0902f3-plugins-config-volume\") pod \"4a67bc25-5eff-4d3f-891b-f377ba0902f3\" (UID: \"4a67bc25-5eff-4d3f-891b-f377ba0902f3\") "
Oct 30 18:49:44.647976 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.647952    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a67bc25-5eff-4d3f-891b-f377ba0902f3-plugins-config-volume" pod "4a67bc25-5eff-4d3f-891b-f377ba0902f3" (UID: "4a67bc25-5eff-4d3f-891b-f377ba0902f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:49:44.648277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.648229    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a67bc25-5eff-4d3f-891b-f377ba0902f3-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:44.649747 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.649728    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a67bc25-5eff-4d3f-891b-f377ba0902f3-kube-api-access-8gns9" pod "4a67bc25-5eff-4d3f-891b-f377ba0902f3" (UID: "4a67bc25-5eff-4d3f-891b-f377ba0902f3"). InnerVolumeSpecName "kube-api-access-8gns9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:49:44.715875 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.715810    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69" podStartSLOduration=2.715789784 podStartE2EDuration="2.715789784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:44.081665049 +0000 UTC m=+997.950083380" watchObservedRunningTime="2026-10-30 18:49:44.715789784 +0000 UTC m=+998.584208115"
Oct 30 18:49:44.751992 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.749216    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5657423d-625a-4c36-9767-6427619d7a50-kube-api-access-smmvh\" (UniqueName: \"kubernetes.io/projected/5657423d-625a-4c36-9767-6427619d7a50-kube-api-access-smmvh\") pod \"5657423d-625a-4c36-9767-6427619d7a50\" (UID: \"5657423d-625a-4c36-9767-6427619d7a50\") "
Oct 30 18:49:44.751992 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.749354    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5657423d-625a-4c36-9767-6427619d7a50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5657423d-625a-4c36-9767-6427619d7a50-plugins-config-volume\") pod \"5657423d-625a-4c36-9767-6427619d7a50\" (UID: \"5657423d-625a-4c36-9767-6427619d7a50\") "
Oct 30 18:49:44.751992 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.750192    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gns9\" (UniqueName: \"kubernetes.io/projected/4a67bc25-5eff-4d3f-891b-f377ba0902f3-kube-api-access-8gns9\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:44.751992 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.750652    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5657423d-625a-4c36-9767-6427619d7a50-plugins-config-volume" pod "5657423d-625a-4c36-9767-6427619d7a50" (UID: "5657423d-625a-4c36-9767-6427619d7a50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:49:44.753476 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.753447    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5657423d-625a-4c36-9767-6427619d7a50-kube-api-access-smmvh" pod "5657423d-625a-4c36-9767-6427619d7a50" (UID: "5657423d-625a-4c36-9767-6427619d7a50"). InnerVolumeSpecName "kube-api-access-smmvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:49:44.852004 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.851976    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smmvh\" (UniqueName: \"kubernetes.io/projected/5657423d-625a-4c36-9767-6427619d7a50-kube-api-access-smmvh\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:44.852004 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:44.852002    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5657423d-625a-4c36-9767-6427619d7a50-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:45.033815 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.033778    4372 generic.go:356] "Generic (PLEG): container finished" podID="5657423d-625a-4c36-9767-6427619d7a50" containerID="67b19d9f93636845ce9ccb77232b2e04b3107b4675074ab8e2b44df6dc5151ca" exitCode=0
Oct 30 18:49:45.034331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.033829    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"
Oct 30 18:49:45.034331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.033860    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl" event={"ID":"5657423d-625a-4c36-9767-6427619d7a50","Type":"ContainerDied","Data":"67b19d9f93636845ce9ccb77232b2e04b3107b4675074ab8e2b44df6dc5151ca"}
Oct 30 18:49:45.034331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.033915    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl" event={"ID":"5657423d-625a-4c36-9767-6427619d7a50","Type":"ContainerDied","Data":"3c5dfe148fe8da76754266df63ad502b718cea9f10848c6edf52f1b90576d0d9"}
Oct 30 18:49:45.034331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.033945    4372 scope.go:122] "RemoveContainer" containerID="67b19d9f93636845ce9ccb77232b2e04b3107b4675074ab8e2b44df6dc5151ca"
Oct 30 18:49:45.035565 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.035533    4372 generic.go:356] "Generic (PLEG): container finished" podID="4a67bc25-5eff-4d3f-891b-f377ba0902f3" containerID="cf1cc81f5fa978c9074cb3d2d9ce98b767fc98ba73c2699563bd48a25d3d5b3f" exitCode=0
Oct 30 18:49:45.035804 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.035586    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"
Oct 30 18:49:45.035804 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.035629    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4" event={"ID":"4a67bc25-5eff-4d3f-891b-f377ba0902f3","Type":"ContainerDied","Data":"cf1cc81f5fa978c9074cb3d2d9ce98b767fc98ba73c2699563bd48a25d3d5b3f"}
Oct 30 18:49:45.035804 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.035660    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4" event={"ID":"4a67bc25-5eff-4d3f-891b-f377ba0902f3","Type":"ContainerDied","Data":"92a4193624d24aa017aae3ea5b28e7c95404538fdfc552354c8faa43c2cf53e5"}
Oct 30 18:49:45.045262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.045245    4372 scope.go:122] "RemoveContainer" containerID="67b19d9f93636845ce9ccb77232b2e04b3107b4675074ab8e2b44df6dc5151ca"
Oct 30 18:49:45.045826 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:49:45.045671    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67b19d9f93636845ce9ccb77232b2e04b3107b4675074ab8e2b44df6dc5151ca\": container with ID starting with 67b19d9f93636845ce9ccb77232b2e04b3107b4675074ab8e2b44df6dc5151ca not found: ID does not exist" containerID="67b19d9f93636845ce9ccb77232b2e04b3107b4675074ab8e2b44df6dc5151ca"
Oct 30 18:49:45.045826 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.045703    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67b19d9f93636845ce9ccb77232b2e04b3107b4675074ab8e2b44df6dc5151ca"} err="failed to get container status \"67b19d9f93636845ce9ccb77232b2e04b3107b4675074ab8e2b44df6dc5151ca\": rpc error: code = NotFound desc = could not find container \"67b19d9f93636845ce9ccb77232b2e04b3107b4675074ab8e2b44df6dc5151ca\": container with ID starting with 67b19d9f93636845ce9ccb77232b2e04b3107b4675074ab8e2b44df6dc5151ca not found: ID does not exist"
Oct 30 18:49:45.045826 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.045729    4372 scope.go:122] "RemoveContainer" containerID="cf1cc81f5fa978c9074cb3d2d9ce98b767fc98ba73c2699563bd48a25d3d5b3f"
Oct 30 18:49:45.069587 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.069520    4372 scope.go:122] "RemoveContainer" containerID="cf1cc81f5fa978c9074cb3d2d9ce98b767fc98ba73c2699563bd48a25d3d5b3f"
Oct 30 18:49:45.069778 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:49:45.069761    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1cc81f5fa978c9074cb3d2d9ce98b767fc98ba73c2699563bd48a25d3d5b3f\": container with ID starting with cf1cc81f5fa978c9074cb3d2d9ce98b767fc98ba73c2699563bd48a25d3d5b3f not found: ID does not exist" containerID="cf1cc81f5fa978c9074cb3d2d9ce98b767fc98ba73c2699563bd48a25d3d5b3f"
Oct 30 18:49:45.069819 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.069784    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1cc81f5fa978c9074cb3d2d9ce98b767fc98ba73c2699563bd48a25d3d5b3f"} err="failed to get container status \"cf1cc81f5fa978c9074cb3d2d9ce98b767fc98ba73c2699563bd48a25d3d5b3f\": rpc error: code = NotFound desc = could not find container \"cf1cc81f5fa978c9074cb3d2d9ce98b767fc98ba73c2699563bd48a25d3d5b3f\": container with ID starting with cf1cc81f5fa978c9074cb3d2d9ce98b767fc98ba73c2699563bd48a25d3d5b3f not found: ID does not exist"
Oct 30 18:49:45.130518 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.130417    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"]
Oct 30 18:49:45.134076 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.134046    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4d94b-f644cd564-lrpdl"]
Oct 30 18:49:45.137828 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.137807    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"]
Oct 30 18:49:45.141160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.141141    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4d94b-746d5d544c-p5fj4"]
Oct 30 18:49:45.242151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.234564    4372 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 18:49:45.242151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.234622    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69" podUID="dcdfd538-a850-48e9-883f-f7428503b8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 18:49:45.422254 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.421955    4372 patch_prober.go:29] interesting pod/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 18:49:45.422254 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:45.422022    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9" podUID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 18:49:46.235172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:46.235113    4372 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 18:49:46.235904 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:46.235871    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69" podUID="dcdfd538-a850-48e9-883f-f7428503b8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 18:49:46.342532 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:46.342493    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a67bc25-5eff-4d3f-891b-f377ba0902f3" path="/var/lib/kubelet/pods/4a67bc25-5eff-4d3f-891b-f377ba0902f3/volumes"
Oct 30 18:49:46.344554 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:46.344528    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5657423d-625a-4c36-9767-6427619d7a50" path="/var/lib/kubelet/pods/5657423d-625a-4c36-9767-6427619d7a50/volumes"
Oct 30 18:49:46.406899 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:46.406867    4372 patch_prober.go:29] interesting pod/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 18:49:46.407179 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:46.407146    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9" podUID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 18:49:47.235151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:47.234917    4372 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 18:49:47.235151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:47.234996    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69" podUID="dcdfd538-a850-48e9-883f-f7428503b8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 18:49:47.401150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:47.400572    4372 patch_prober.go:29] interesting pod/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 18:49:47.401150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:47.400630    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9" podUID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 18:49:48.232135 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:48.232102    4372 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 18:49:48.232319 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:48.232181    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69" podUID="dcdfd538-a850-48e9-883f-f7428503b8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 18:49:48.441459 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:48.441428    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:49:48.443588 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:48.443566    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:49:48.448148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:48.445619    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:49:49.231436 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:49.231402    4372 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 18:49:49.231615 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:49.231457    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69" podUID="dcdfd538-a850-48e9-883f-f7428503b8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 18:49:50.231417 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:50.231379    4372 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 18:49:50.231816 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:50.231435    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69" podUID="dcdfd538-a850-48e9-883f-f7428503b8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 18:49:51.300939 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:51.300890    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:51.303446 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:51.303423    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:51.304930 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:51.304907    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:49:52.977146 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:52.977003    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"]
Oct 30 18:49:52.977631 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:52.977346    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp" podUID="f4b5c108-eede-4a72-8335-2751062c90ec" containerName="maas-api" containerID="cri-o://793782c71835ebbd0645c9ca9d6c7cf2071a97220bfc709b8fc7a225539b0226" gracePeriod=30
Oct 30 18:49:53.541419 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:53.540345    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:49:53.631255 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:53.631172    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b5c108-eede-4a72-8335-2751062c90ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b5c108-eede-4a72-8335-2751062c90ec-openshift-service-ca\") pod \"f4b5c108-eede-4a72-8335-2751062c90ec\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") "
Oct 30 18:49:53.631255 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:53.631236    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4b5c108-eede-4a72-8335-2751062c90ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b5c108-eede-4a72-8335-2751062c90ec-metrics-certs\") pod \"f4b5c108-eede-4a72-8335-2751062c90ec\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") "
Oct 30 18:49:53.631436 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:53.631357    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4b5c108-eede-4a72-8335-2751062c90ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b5c108-eede-4a72-8335-2751062c90ec-maas-api-tls\") pod \"f4b5c108-eede-4a72-8335-2751062c90ec\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") "
Oct 30 18:49:53.631469 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:53.631439    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b5c108-eede-4a72-8335-2751062c90ec-kube-api-access-kc8hx\" (UniqueName: \"kubernetes.io/projected/f4b5c108-eede-4a72-8335-2751062c90ec-kube-api-access-kc8hx\") pod \"f4b5c108-eede-4a72-8335-2751062c90ec\" (UID: \"f4b5c108-eede-4a72-8335-2751062c90ec\") "
Oct 30 18:49:53.631636 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:53.631613    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b5c108-eede-4a72-8335-2751062c90ec-openshift-service-ca" pod "f4b5c108-eede-4a72-8335-2751062c90ec" (UID: "f4b5c108-eede-4a72-8335-2751062c90ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:49:53.632247 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:53.632152    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b5c108-eede-4a72-8335-2751062c90ec-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:53.633637 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:53.633607    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b5c108-eede-4a72-8335-2751062c90ec-metrics-certs" pod "f4b5c108-eede-4a72-8335-2751062c90ec" (UID: "f4b5c108-eede-4a72-8335-2751062c90ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:49:53.633637 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:53.633625    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b5c108-eede-4a72-8335-2751062c90ec-maas-api-tls" pod "f4b5c108-eede-4a72-8335-2751062c90ec" (UID: "f4b5c108-eede-4a72-8335-2751062c90ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:49:53.633761 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:53.633683    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b5c108-eede-4a72-8335-2751062c90ec-kube-api-access-kc8hx" pod "f4b5c108-eede-4a72-8335-2751062c90ec" (UID: "f4b5c108-eede-4a72-8335-2751062c90ec"). InnerVolumeSpecName "kube-api-access-kc8hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:49:53.738152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:53.735271    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc8hx\" (UniqueName: \"kubernetes.io/projected/f4b5c108-eede-4a72-8335-2751062c90ec-kube-api-access-kc8hx\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:53.738152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:53.735314    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b5c108-eede-4a72-8335-2751062c90ec-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:53.738152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:53.735327    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b5c108-eede-4a72-8335-2751062c90ec-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:54.101414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:54.101382    4372 generic.go:356] "Generic (PLEG): container finished" podID="f4b5c108-eede-4a72-8335-2751062c90ec" containerID="793782c71835ebbd0645c9ca9d6c7cf2071a97220bfc709b8fc7a225539b0226" exitCode=0
Oct 30 18:49:54.101822 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:54.101428    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"
Oct 30 18:49:54.101822 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:54.101463    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp" event={"ID":"f4b5c108-eede-4a72-8335-2751062c90ec","Type":"ContainerDied","Data":"793782c71835ebbd0645c9ca9d6c7cf2071a97220bfc709b8fc7a225539b0226"}
Oct 30 18:49:54.101822 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:54.101562    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp" event={"ID":"f4b5c108-eede-4a72-8335-2751062c90ec","Type":"ContainerDied","Data":"a03eb6ef001f3afb80f377aa8525ed91c3f778bf11e8e78766936a996e6e4786"}
Oct 30 18:49:54.101822 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:54.101597    4372 scope.go:122] "RemoveContainer" containerID="793782c71835ebbd0645c9ca9d6c7cf2071a97220bfc709b8fc7a225539b0226"
Oct 30 18:49:54.112575 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:54.112556    4372 scope.go:122] "RemoveContainer" containerID="793782c71835ebbd0645c9ca9d6c7cf2071a97220bfc709b8fc7a225539b0226"
Oct 30 18:49:54.112865 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:49:54.112847    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"793782c71835ebbd0645c9ca9d6c7cf2071a97220bfc709b8fc7a225539b0226\": container with ID starting with 793782c71835ebbd0645c9ca9d6c7cf2071a97220bfc709b8fc7a225539b0226 not found: ID does not exist" containerID="793782c71835ebbd0645c9ca9d6c7cf2071a97220bfc709b8fc7a225539b0226"
Oct 30 18:49:54.112918 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:54.112874    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"793782c71835ebbd0645c9ca9d6c7cf2071a97220bfc709b8fc7a225539b0226"} err="failed to get container status \"793782c71835ebbd0645c9ca9d6c7cf2071a97220bfc709b8fc7a225539b0226\": rpc error: code = NotFound desc = could not find container \"793782c71835ebbd0645c9ca9d6c7cf2071a97220bfc709b8fc7a225539b0226\": container with ID starting with 793782c71835ebbd0645c9ca9d6c7cf2071a97220bfc709b8fc7a225539b0226 not found: ID does not exist"
Oct 30 18:49:54.128589 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:54.128570    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"]
Oct 30 18:49:54.132383 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:54.132360    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4d94b-755d6d5ccc-5tvsp"]
Oct 30 18:49:54.266717 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:49:54.266648    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b5c108-eede-4a72-8335-2751062c90ec" path="/var/lib/kubelet/pods/f4b5c108-eede-4a72-8335-2751062c90ec/volumes"
Oct 30 18:49:57.696783 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:49:57.696744    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:50:02.582759 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:02.582726    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"]
Oct 30 18:50:07.583438 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:07.583383    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn" podUID="02abb3d7-836b-46e7-b2d1-b130e2a60bfe" containerName="payload-pre-processing" containerID="cri-o://141c44acdc6b6ec19386c8baaee25b7ed284377137da40e44484ef748e6a4575" gracePeriod=25
Oct 30 18:50:07.870460 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:07.870436    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"
Oct 30 18:50:08.023326 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:08.023252    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02abb3d7-836b-46e7-b2d1-b130e2a60bfe-kube-api-access-9qj6b\" (UniqueName: \"kubernetes.io/projected/02abb3d7-836b-46e7-b2d1-b130e2a60bfe-kube-api-access-9qj6b\") pod \"02abb3d7-836b-46e7-b2d1-b130e2a60bfe\" (UID: \"02abb3d7-836b-46e7-b2d1-b130e2a60bfe\") "
Oct 30 18:50:08.023469 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:08.023381    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02abb3d7-836b-46e7-b2d1-b130e2a60bfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02abb3d7-836b-46e7-b2d1-b130e2a60bfe-plugins-config-volume\") pod \"02abb3d7-836b-46e7-b2d1-b130e2a60bfe\" (UID: \"02abb3d7-836b-46e7-b2d1-b130e2a60bfe\") "
Oct 30 18:50:08.023794 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:08.023771    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02abb3d7-836b-46e7-b2d1-b130e2a60bfe-plugins-config-volume" pod "02abb3d7-836b-46e7-b2d1-b130e2a60bfe" (UID: "02abb3d7-836b-46e7-b2d1-b130e2a60bfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:08.024005 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:08.023980    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02abb3d7-836b-46e7-b2d1-b130e2a60bfe-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:08.025515 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:08.025491    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02abb3d7-836b-46e7-b2d1-b130e2a60bfe-kube-api-access-9qj6b" pod "02abb3d7-836b-46e7-b2d1-b130e2a60bfe" (UID: "02abb3d7-836b-46e7-b2d1-b130e2a60bfe"). InnerVolumeSpecName "kube-api-access-9qj6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:08.125172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:08.125111    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qj6b\" (UniqueName: \"kubernetes.io/projected/02abb3d7-836b-46e7-b2d1-b130e2a60bfe-kube-api-access-9qj6b\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:08.539946 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:08.539290    4372 generic.go:356] "Generic (PLEG): container finished" podID="02abb3d7-836b-46e7-b2d1-b130e2a60bfe" containerID="141c44acdc6b6ec19386c8baaee25b7ed284377137da40e44484ef748e6a4575" exitCode=0
Oct 30 18:50:08.539946 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:08.539376    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn" event={"ID":"02abb3d7-836b-46e7-b2d1-b130e2a60bfe","Type":"ContainerDied","Data":"141c44acdc6b6ec19386c8baaee25b7ed284377137da40e44484ef748e6a4575"}
Oct 30 18:50:08.539946 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:08.539432    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn" event={"ID":"02abb3d7-836b-46e7-b2d1-b130e2a60bfe","Type":"ContainerDied","Data":"1bb4c9e29872d6979556572a629b03dc2e0e57062572cc544c67165e76edd72b"}
Oct 30 18:50:08.539946 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:08.539463    4372 scope.go:122] "RemoveContainer" containerID="141c44acdc6b6ec19386c8baaee25b7ed284377137da40e44484ef748e6a4575"
Oct 30 18:50:08.539946 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:08.539645    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"
Oct 30 18:50:08.554722 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:08.554699    4372 scope.go:122] "RemoveContainer" containerID="141c44acdc6b6ec19386c8baaee25b7ed284377137da40e44484ef748e6a4575"
Oct 30 18:50:08.555047 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:50:08.555023    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141c44acdc6b6ec19386c8baaee25b7ed284377137da40e44484ef748e6a4575\": container with ID starting with 141c44acdc6b6ec19386c8baaee25b7ed284377137da40e44484ef748e6a4575 not found: ID does not exist" containerID="141c44acdc6b6ec19386c8baaee25b7ed284377137da40e44484ef748e6a4575"
Oct 30 18:50:08.555182 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:08.555054    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141c44acdc6b6ec19386c8baaee25b7ed284377137da40e44484ef748e6a4575"} err="failed to get container status \"141c44acdc6b6ec19386c8baaee25b7ed284377137da40e44484ef748e6a4575\": rpc error: code = NotFound desc = could not find container \"141c44acdc6b6ec19386c8baaee25b7ed284377137da40e44484ef748e6a4575\": container with ID starting with 141c44acdc6b6ec19386c8baaee25b7ed284377137da40e44484ef748e6a4575 not found: ID does not exist"
Oct 30 18:50:08.564285 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:08.564262    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"]
Oct 30 18:50:08.567673 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:08.567653    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4b265-76c66c5c9-7tvcn"]
Oct 30 18:50:10.266466 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:10.266434    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02abb3d7-836b-46e7-b2d1-b130e2a60bfe" path="/var/lib/kubelet/pods/02abb3d7-836b-46e7-b2d1-b130e2a60bfe/volumes"
Oct 30 18:50:11.048694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.048665    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7411-6148bf"]
Oct 30 18:50:11.049301 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.049282    4372 cpu_manager.go:414] "containerMap: removing container" podUID="4a67bc25-5eff-4d3f-891b-f377ba0902f3" containerName="payload-pre-processing"
Oct 30 18:50:11.049301 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.049298    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a67bc25-5eff-4d3f-891b-f377ba0902f3" containerName="payload-pre-processing"
Oct 30 18:50:11.049467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.049309    4372 cpu_manager.go:414] "containerMap: removing container" podUID="02abb3d7-836b-46e7-b2d1-b130e2a60bfe" containerName="payload-pre-processing"
Oct 30 18:50:11.049467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.049314    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02abb3d7-836b-46e7-b2d1-b130e2a60bfe" containerName="payload-pre-processing"
Oct 30 18:50:11.049467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.049327    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f4b5c108-eede-4a72-8335-2751062c90ec" containerName="maas-api"
Oct 30 18:50:11.049467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.049332    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b5c108-eede-4a72-8335-2751062c90ec" containerName="maas-api"
Oct 30 18:50:11.049467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.049341    4372 cpu_manager.go:414] "containerMap: removing container" podUID="5657423d-625a-4c36-9767-6427619d7a50" containerName="payload-processing"
Oct 30 18:50:11.049467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.049346    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5657423d-625a-4c36-9767-6427619d7a50" containerName="payload-processing"
Oct 30 18:50:11.049467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.049453    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a67bc25-5eff-4d3f-891b-f377ba0902f3" containerName="payload-pre-processing"
Oct 30 18:50:11.049467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.049460    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="02abb3d7-836b-46e7-b2d1-b130e2a60bfe" containerName="payload-pre-processing"
Oct 30 18:50:11.049467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.049468    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b5c108-eede-4a72-8335-2751062c90ec" containerName="maas-api"
Oct 30 18:50:11.049739 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.049474    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="5657423d-625a-4c36-9767-6427619d7a50" containerName="payload-processing"
Oct 30 18:50:11.054404 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.054387    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7411-6148bf"
Oct 30 18:50:11.060425 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.059702    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7h66b\""
Oct 30 18:50:11.061322 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.061299    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 18:50:11.061322 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.061301    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 18:50:11.062404 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.062388    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7411-6148bf"]
Oct 30 18:50:11.090382 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.090350    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkh5l\" (UniqueName: \"kubernetes.io/projected/82c34738-0746-4ea1-840e-3ab7514980c9-kube-api-access-mkh5l\") pod \"test-curl-7411-6148bf\" (UID: \"82c34738-0746-4ea1-840e-3ab7514980c9\") " pod="opendatahub/test-curl-7411-6148bf"
Oct 30 18:50:11.191528 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.191498    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkh5l\" (UniqueName: \"kubernetes.io/projected/82c34738-0746-4ea1-840e-3ab7514980c9-kube-api-access-mkh5l\") pod \"test-curl-7411-6148bf\" (UID: \"82c34738-0746-4ea1-840e-3ab7514980c9\") " pod="opendatahub/test-curl-7411-6148bf"
Oct 30 18:50:11.206111 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.206088    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkh5l\" (UniqueName: \"kubernetes.io/projected/82c34738-0746-4ea1-840e-3ab7514980c9-kube-api-access-mkh5l\") pod \"test-curl-7411-6148bf\" (UID: \"82c34738-0746-4ea1-840e-3ab7514980c9\") " pod="opendatahub/test-curl-7411-6148bf"
Oct 30 18:50:11.366766 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.366738    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"]
Oct 30 18:50:11.367256 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.366958    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb" podUID="8745c459-008e-471d-a4b4-67da58c8432d" containerName="maas-api" containerID="cri-o://5103b96bd8a32cdebe35fc60691f9d9adb7685067c32e8e71fc5f37f3bf28a99" gracePeriod=30
Oct 30 18:50:11.369789 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.369762    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7411-6148bf"
Oct 30 18:50:11.683201 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.681117    4372 generic.go:356] "Generic (PLEG): container finished" podID="8745c459-008e-471d-a4b4-67da58c8432d" containerID="5103b96bd8a32cdebe35fc60691f9d9adb7685067c32e8e71fc5f37f3bf28a99" exitCode=0
Oct 30 18:50:11.683201 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.681359    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb" event={"ID":"8745c459-008e-471d-a4b4-67da58c8432d","Type":"ContainerDied","Data":"5103b96bd8a32cdebe35fc60691f9d9adb7685067c32e8e71fc5f37f3bf28a99"}
Oct 30 18:50:11.775173 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.767440    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7411-6148bf"]
Oct 30 18:50:11.867829 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:50:11.867770    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82c34738_0746_4ea1_840e_3ab7514980c9.slice/crio-9ead33651b2e8020877f2d1ace994138fa4d579bfae485b8e22950aaef75bef1 WatchSource:0}: Error finding container 9ead33651b2e8020877f2d1ace994138fa4d579bfae485b8e22950aaef75bef1: Status 404 returned error can't find the container with id 9ead33651b2e8020877f2d1ace994138fa4d579bfae485b8e22950aaef75bef1
Oct 30 18:50:11.929963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:11.929939    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:50:12.029520 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.029444    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"]
Oct 30 18:50:12.032759 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.032738    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dbdac51-gmskc"]
Oct 30 18:50:12.071333 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.071299    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8745c459-008e-471d-a4b4-67da58c8432d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8745c459-008e-471d-a4b4-67da58c8432d-metrics-certs\") pod \"8745c459-008e-471d-a4b4-67da58c8432d\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") "
Oct 30 18:50:12.071721 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.071698    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8745c459-008e-471d-a4b4-67da58c8432d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8745c459-008e-471d-a4b4-67da58c8432d-openshift-service-ca\") pod \"8745c459-008e-471d-a4b4-67da58c8432d\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") "
Oct 30 18:50:12.071894 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.071879    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8745c459-008e-471d-a4b4-67da58c8432d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8745c459-008e-471d-a4b4-67da58c8432d-maas-api-tls\") pod \"8745c459-008e-471d-a4b4-67da58c8432d\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") "
Oct 30 18:50:12.090191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.079734    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8745c459-008e-471d-a4b4-67da58c8432d-kube-api-access-z2lg7\" (UniqueName: \"kubernetes.io/projected/8745c459-008e-471d-a4b4-67da58c8432d-kube-api-access-z2lg7\") pod \"8745c459-008e-471d-a4b4-67da58c8432d\" (UID: \"8745c459-008e-471d-a4b4-67da58c8432d\") "
Oct 30 18:50:12.097326 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.097276    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8745c459-008e-471d-a4b4-67da58c8432d-maas-api-tls" pod "8745c459-008e-471d-a4b4-67da58c8432d" (UID: "8745c459-008e-471d-a4b4-67da58c8432d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:50:12.097719 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.097696    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8745c459-008e-471d-a4b4-67da58c8432d-openshift-service-ca" pod "8745c459-008e-471d-a4b4-67da58c8432d" (UID: "8745c459-008e-471d-a4b4-67da58c8432d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:12.097976 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.097926    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8745c459-008e-471d-a4b4-67da58c8432d-kube-api-access-z2lg7" pod "8745c459-008e-471d-a4b4-67da58c8432d" (UID: "8745c459-008e-471d-a4b4-67da58c8432d"). InnerVolumeSpecName "kube-api-access-z2lg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:12.102331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.102299    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8745c459-008e-471d-a4b4-67da58c8432d-metrics-certs" pod "8745c459-008e-471d-a4b4-67da58c8432d" (UID: "8745c459-008e-471d-a4b4-67da58c8432d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:50:12.196088 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.196032    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8745c459-008e-471d-a4b4-67da58c8432d-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:12.196088 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.196068    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8745c459-008e-471d-a4b4-67da58c8432d-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:12.196088 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.196085    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8745c459-008e-471d-a4b4-67da58c8432d-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:12.196353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.196098    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2lg7\" (UniqueName: \"kubernetes.io/projected/8745c459-008e-471d-a4b4-67da58c8432d-kube-api-access-z2lg7\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:12.265730 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.265703    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9fa7e43-0cc5-4520-8d34-e20e611b5fbc" path="/var/lib/kubelet/pods/a9fa7e43-0cc5-4520-8d34-e20e611b5fbc/volumes"
Oct 30 18:50:12.347470 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.347380    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"]
Oct 30 18:50:12.788731 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.788647    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb" event={"ID":"8745c459-008e-471d-a4b4-67da58c8432d","Type":"ContainerDied","Data":"1bdda4d9cb4ed6b2c610885da341ca5346c3719fc7d63ee194f0d7de297a1fdb"}
Oct 30 18:50:12.796331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.796305    4372 scope.go:122] "RemoveContainer" containerID="5103b96bd8a32cdebe35fc60691f9d9adb7685067c32e8e71fc5f37f3bf28a99"
Oct 30 18:50:12.796639 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.796623    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"
Oct 30 18:50:12.826873 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.826177    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7411-6148bf" event={"ID":"82c34738-0746-4ea1-840e-3ab7514980c9","Type":"ContainerStarted","Data":"9ead33651b2e8020877f2d1ace994138fa4d579bfae485b8e22950aaef75bef1"}
Oct 30 18:50:12.866148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.865648    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"]
Oct 30 18:50:12.886875 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:12.886837    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4b265-74bbf5df97-kjsfb"]
Oct 30 18:50:14.268953 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:14.268924    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8745c459-008e-471d-a4b4-67da58c8432d" path="/var/lib/kubelet/pods/8745c459-008e-471d-a4b4-67da58c8432d/volumes"
Oct 30 18:50:14.573045 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:14.572969    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:50:14.906150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:14.904826    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7411-6148bf" event={"ID":"82c34738-0746-4ea1-840e-3ab7514980c9","Type":"ContainerStarted","Data":"649c969eadd8fc88dd455af2d30dd6e838e8855d9b6b35b47aa3741b1dc3c950"}
Oct 30 18:50:14.941192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:14.938829    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7411-6148bf" podStartSLOduration=1.128740935 podStartE2EDuration="3.938797244s" totalImagesPullingTime="2.810056309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:14.93418631 +0000 UTC m=+1028.802604644" watchObservedRunningTime="2026-10-30 18:50:14.938797244 +0000 UTC m=+1028.807215575"
Oct 30 18:50:15.738153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:15.735221    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7411-6148bf"]
Oct 30 18:50:15.796147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:15.747357    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7411-6148bf"]
Oct 30 18:50:17.012286 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:17.012205    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7411-6148bf" podUID="82c34738-0746-4ea1-840e-3ab7514980c9" containerName="test-curl-7411-6148bf" containerID="cri-o://649c969eadd8fc88dd455af2d30dd6e838e8855d9b6b35b47aa3741b1dc3c950" gracePeriod=2
Oct 30 18:50:17.075226 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:17.075189    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"]
Oct 30 18:50:17.092191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:17.080108    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02159ea4-tvqbp"]
Oct 30 18:50:17.347737 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:17.347689    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk" podUID="01404f15-9f9f-446c-88d0-83e2f67bd877" containerName="payload-processing" containerID="cri-o://a5c86532cadab9838b4d4da0117400fabeec91a3ebf1888518e9aefc84ec1149" gracePeriod=25
Oct 30 18:50:17.627660 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:17.627637    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"
Oct 30 18:50:17.726737 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:17.726708    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01404f15-9f9f-446c-88d0-83e2f67bd877-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01404f15-9f9f-446c-88d0-83e2f67bd877-plugins-config-volume\") pod \"01404f15-9f9f-446c-88d0-83e2f67bd877\" (UID: \"01404f15-9f9f-446c-88d0-83e2f67bd877\") "
Oct 30 18:50:17.726935 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:17.726916    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01404f15-9f9f-446c-88d0-83e2f67bd877-kube-api-access-tqbbd\" (UniqueName: \"kubernetes.io/projected/01404f15-9f9f-446c-88d0-83e2f67bd877-kube-api-access-tqbbd\") pod \"01404f15-9f9f-446c-88d0-83e2f67bd877\" (UID: \"01404f15-9f9f-446c-88d0-83e2f67bd877\") "
Oct 30 18:50:17.727147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:17.727107    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01404f15-9f9f-446c-88d0-83e2f67bd877-plugins-config-volume" pod "01404f15-9f9f-446c-88d0-83e2f67bd877" (UID: "01404f15-9f9f-446c-88d0-83e2f67bd877"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:17.727677 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:17.727557    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01404f15-9f9f-446c-88d0-83e2f67bd877-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:17.729377 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:17.729353    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01404f15-9f9f-446c-88d0-83e2f67bd877-kube-api-access-tqbbd" pod "01404f15-9f9f-446c-88d0-83e2f67bd877" (UID: "01404f15-9f9f-446c-88d0-83e2f67bd877"). InnerVolumeSpecName "kube-api-access-tqbbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:17.830803 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:17.829766    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqbbd\" (UniqueName: \"kubernetes.io/projected/01404f15-9f9f-446c-88d0-83e2f67bd877-kube-api-access-tqbbd\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:18.088203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:18.068724    4372 generic.go:356] "Generic (PLEG): container finished" podID="01404f15-9f9f-446c-88d0-83e2f67bd877" containerID="a5c86532cadab9838b4d4da0117400fabeec91a3ebf1888518e9aefc84ec1149" exitCode=0
Oct 30 18:50:18.088203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:18.068866    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk" event={"ID":"01404f15-9f9f-446c-88d0-83e2f67bd877","Type":"ContainerDied","Data":"a5c86532cadab9838b4d4da0117400fabeec91a3ebf1888518e9aefc84ec1149"}
Oct 30 18:50:18.088203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:18.068946    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk" event={"ID":"01404f15-9f9f-446c-88d0-83e2f67bd877","Type":"ContainerDied","Data":"4af442e5388a21f04c72bc5a562decd841f03aa4be3b83e41a216a74f7cfe17f"}
Oct 30 18:50:18.088203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:18.068982    4372 scope.go:122] "RemoveContainer" containerID="a5c86532cadab9838b4d4da0117400fabeec91a3ebf1888518e9aefc84ec1149"
Oct 30 18:50:18.088203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:18.069193    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"
Oct 30 18:50:18.182382 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:18.182351    4372 scope.go:122] "RemoveContainer" containerID="a5c86532cadab9838b4d4da0117400fabeec91a3ebf1888518e9aefc84ec1149"
Oct 30 18:50:18.186167 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:50:18.183091    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c86532cadab9838b4d4da0117400fabeec91a3ebf1888518e9aefc84ec1149\": container with ID starting with a5c86532cadab9838b4d4da0117400fabeec91a3ebf1888518e9aefc84ec1149 not found: ID does not exist" containerID="a5c86532cadab9838b4d4da0117400fabeec91a3ebf1888518e9aefc84ec1149"
Oct 30 18:50:18.194149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:18.191211    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c86532cadab9838b4d4da0117400fabeec91a3ebf1888518e9aefc84ec1149"} err="failed to get container status \"a5c86532cadab9838b4d4da0117400fabeec91a3ebf1888518e9aefc84ec1149\": rpc error: code = NotFound desc = could not find container \"a5c86532cadab9838b4d4da0117400fabeec91a3ebf1888518e9aefc84ec1149\": container with ID starting with a5c86532cadab9838b4d4da0117400fabeec91a3ebf1888518e9aefc84ec1149 not found: ID does not exist"
Oct 30 18:50:18.211988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:18.210803    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"]
Oct 30 18:50:18.211988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:18.210869    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f44051-7cdfc8f588-2gtsk"]
Oct 30 18:50:18.282496 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:18.282458    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01404f15-9f9f-446c-88d0-83e2f67bd877" path="/var/lib/kubelet/pods/01404f15-9f9f-446c-88d0-83e2f67bd877/volumes"
Oct 30 18:50:18.283157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:18.283115    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1581874-75a0-40ca-a7b8-0dc3a854b300" path="/var/lib/kubelet/pods/a1581874-75a0-40ca-a7b8-0dc3a854b300/volumes"
Oct 30 18:50:19.079492 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:19.079467    4372 generic.go:356] "Generic (PLEG): container finished" podID="82c34738-0746-4ea1-840e-3ab7514980c9" containerID="649c969eadd8fc88dd455af2d30dd6e838e8855d9b6b35b47aa3741b1dc3c950" exitCode=137
Oct 30 18:50:19.289778 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:19.289750    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7411-6148bf"
Oct 30 18:50:19.417893 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:19.417847    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82c34738-0746-4ea1-840e-3ab7514980c9-kube-api-access-mkh5l\" (UniqueName: \"kubernetes.io/projected/82c34738-0746-4ea1-840e-3ab7514980c9-kube-api-access-mkh5l\") pod \"82c34738-0746-4ea1-840e-3ab7514980c9\" (UID: \"82c34738-0746-4ea1-840e-3ab7514980c9\") "
Oct 30 18:50:19.442167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:19.439301    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c34738-0746-4ea1-840e-3ab7514980c9-kube-api-access-mkh5l" pod "82c34738-0746-4ea1-840e-3ab7514980c9" (UID: "82c34738-0746-4ea1-840e-3ab7514980c9"). InnerVolumeSpecName "kube-api-access-mkh5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:19.489286 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:19.489256    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"]
Oct 30 18:50:19.489496 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:19.489473    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9" podUID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" containerName="istio-proxy" containerID="cri-o://c2bd126e2fec738bc5193b25692603ac8c9d03917373707935590e44f5c2743c" gracePeriod=30
Oct 30 18:50:19.520706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:19.520637    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkh5l\" (UniqueName: \"kubernetes.io/projected/82c34738-0746-4ea1-840e-3ab7514980c9-kube-api-access-mkh5l\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:20.089079 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:20.089054    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7411-6148bf"
Oct 30 18:50:20.089313 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:20.089105    4372 scope.go:122] "RemoveContainer" containerID="649c969eadd8fc88dd455af2d30dd6e838e8855d9b6b35b47aa3741b1dc3c950"
Oct 30 18:50:20.100592 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:50:20.100564    4372 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7411-6148bf\" is forbidden: User \"system:node:ip-10-0-142-153.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-153.us-east-2.compute.internal' and this object" podUID="82c34738-0746-4ea1-840e-3ab7514980c9" pod="opendatahub/test-curl-7411-6148bf"
Oct 30 18:50:20.266442 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:20.266416    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82c34738-0746-4ea1-840e-3ab7514980c9" path="/var/lib/kubelet/pods/82c34738-0746-4ea1-840e-3ab7514980c9/volumes"
Oct 30 18:50:24.764659 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.764605    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:50:24.827966 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.827931    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-envoy\") pod \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") "
Oct 30 18:50:24.828144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.828003    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-kube-api-access-2wrws\" (UniqueName: \"kubernetes.io/projected/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-kube-api-access-2wrws\") pod \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") "
Oct 30 18:50:24.828144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.828060    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-data\") pod \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") "
Oct 30 18:50:24.828144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.828140    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istiod-ca-cert\") pod \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") "
Oct 30 18:50:24.828318 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.828244    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-credential-socket\") pod \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") "
Oct 30 18:50:24.828374 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.828319    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-workload-socket\") pod \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") "
Oct 30 18:50:24.828429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.828407    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-podinfo\") pod \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") "
Oct 30 18:50:24.828429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.828406    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-data" pod "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" (UID: "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:24.828525 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.828495    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-workload-certs\") pod \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") "
Oct 30 18:50:24.828578 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.828542    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-token\" (UniqueName: \"kubernetes.io/projected/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-token\") pod \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\" (UID: \"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4\") "
Oct 30 18:50:24.828868 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.828617    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-workload-socket" pod "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" (UID: "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:24.828868 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.828640    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istiod-ca-cert" pod "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" (UID: "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:24.828868 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.828819    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-credential-socket" pod "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" (UID: "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:24.828868 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.828833    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-workload-certs" pod "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" (UID: "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:24.829352 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.829306    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:24.829352 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.829333    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:24.829352 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.829349    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:24.829562 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.829367    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:24.829562 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.829382    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:24.830855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.830832    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-envoy" pod "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" (UID: "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:24.831119 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.831096    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-kube-api-access-2wrws" pod "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" (UID: "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4"). InnerVolumeSpecName "kube-api-access-2wrws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:24.831469 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.831441    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-token" pod "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" (UID: "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:24.831604 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.831582    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-podinfo" pod "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" (UID: "ca9ec230-0cb3-4296-a9f1-58d09a8e17b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:50:24.939274 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.939186    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:24.939274 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.939220    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:24.939274 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.939235    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wrws\" (UniqueName: \"kubernetes.io/projected/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-kube-api-access-2wrws\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:24.939274 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:24.939249    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:25.212285 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:25.212251    4372 generic.go:356] "Generic (PLEG): container finished" podID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" containerID="c2bd126e2fec738bc5193b25692603ac8c9d03917373707935590e44f5c2743c" exitCode=0
Oct 30 18:50:25.212452 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:25.212298    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9" event={"ID":"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4","Type":"ContainerDied","Data":"c2bd126e2fec738bc5193b25692603ac8c9d03917373707935590e44f5c2743c"}
Oct 30 18:50:25.212452 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:25.212316    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"
Oct 30 18:50:25.212452 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:25.212362    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9" event={"ID":"ca9ec230-0cb3-4296-a9f1-58d09a8e17b4","Type":"ContainerDied","Data":"002a4f733558b9dbe4554edcf3145da6cd334bbd1e5266de99fbf596290029b3"}
Oct 30 18:50:25.212452 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:25.212384    4372 scope.go:122] "RemoveContainer" containerID="c2bd126e2fec738bc5193b25692603ac8c9d03917373707935590e44f5c2743c"
Oct 30 18:50:25.223267 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:25.223243    4372 scope.go:122] "RemoveContainer" containerID="317dd8af0e349a3c71a4f6aabfd87950085b9649de056d3d0447170817451d50"
Oct 30 18:50:25.232229 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:25.232213    4372 scope.go:122] "RemoveContainer" containerID="c2bd126e2fec738bc5193b25692603ac8c9d03917373707935590e44f5c2743c"
Oct 30 18:50:25.232510 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:50:25.232488    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2bd126e2fec738bc5193b25692603ac8c9d03917373707935590e44f5c2743c\": container with ID starting with c2bd126e2fec738bc5193b25692603ac8c9d03917373707935590e44f5c2743c not found: ID does not exist" containerID="c2bd126e2fec738bc5193b25692603ac8c9d03917373707935590e44f5c2743c"
Oct 30 18:50:25.232611 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:25.232516    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2bd126e2fec738bc5193b25692603ac8c9d03917373707935590e44f5c2743c"} err="failed to get container status \"c2bd126e2fec738bc5193b25692603ac8c9d03917373707935590e44f5c2743c\": rpc error: code = NotFound desc = could not find container \"c2bd126e2fec738bc5193b25692603ac8c9d03917373707935590e44f5c2743c\": container with ID starting with c2bd126e2fec738bc5193b25692603ac8c9d03917373707935590e44f5c2743c not found: ID does not exist"
Oct 30 18:50:25.232611 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:25.232531    4372 scope.go:122] "RemoveContainer" containerID="317dd8af0e349a3c71a4f6aabfd87950085b9649de056d3d0447170817451d50"
Oct 30 18:50:25.232753 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:50:25.232737    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"317dd8af0e349a3c71a4f6aabfd87950085b9649de056d3d0447170817451d50\": container with ID starting with 317dd8af0e349a3c71a4f6aabfd87950085b9649de056d3d0447170817451d50 not found: ID does not exist" containerID="317dd8af0e349a3c71a4f6aabfd87950085b9649de056d3d0447170817451d50"
Oct 30 18:50:25.232789 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:25.232756    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"317dd8af0e349a3c71a4f6aabfd87950085b9649de056d3d0447170817451d50"} err="failed to get container status \"317dd8af0e349a3c71a4f6aabfd87950085b9649de056d3d0447170817451d50\": rpc error: code = NotFound desc = could not find container \"317dd8af0e349a3c71a4f6aabfd87950085b9649de056d3d0447170817451d50\": container with ID starting with 317dd8af0e349a3c71a4f6aabfd87950085b9649de056d3d0447170817451d50 not found: ID does not exist"
Oct 30 18:50:25.238979 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:25.238947    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"]
Oct 30 18:50:25.242335 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:25.242315    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f4b265-openshift-default-64458fc847-p4dj9"]
Oct 30 18:50:26.268585 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:26.268553    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" path="/var/lib/kubelet/pods/ca9ec230-0cb3-4296-a9f1-58d09a8e17b4/volumes"
Oct 30 18:50:27.695987 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:50:27.695947    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:50:36.319461 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:36.319427    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"]
Oct 30 18:50:36.320192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:36.319651    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69" podUID="dcdfd538-a850-48e9-883f-f7428503b8a9" containerName="istio-proxy" containerID="cri-o://7451ab21d620e6bf75ec88c0997ab82aeaf30f64bea334f9e2a2380b8bd9a87c" gracePeriod=30
Oct 30 18:50:36.728742 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:36.728706    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"]
Oct 30 18:50:41.557396 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.557348    4372 generic.go:356] "Generic (PLEG): container finished" podID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" containerID="665eeabdf2bc826c6ee54928b394063be812396e1cc4002b419620c08d44fb39" exitCode=0
Oct 30 18:50:41.557810 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.557534    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn" event={"ID":"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3","Type":"ContainerDied","Data":"665eeabdf2bc826c6ee54928b394063be812396e1cc4002b419620c08d44fb39"}
Oct 30 18:50:41.558206 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.558096    4372 scope.go:122] "RemoveContainer" containerID="665eeabdf2bc826c6ee54928b394063be812396e1cc4002b419620c08d44fb39"
Oct 30 18:50:41.560570 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.560422    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:50:41.560570 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.560491    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:50:41.560570 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.560523    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:50:41.561938 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.561912    4372 generic.go:356] "Generic (PLEG): container finished" podID="dcdfd538-a850-48e9-883f-f7428503b8a9" containerID="7451ab21d620e6bf75ec88c0997ab82aeaf30f64bea334f9e2a2380b8bd9a87c" exitCode=0
Oct 30 18:50:41.562025 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.561975    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69" event={"ID":"dcdfd538-a850-48e9-883f-f7428503b8a9","Type":"ContainerDied","Data":"7451ab21d620e6bf75ec88c0997ab82aeaf30f64bea334f9e2a2380b8bd9a87c"}
Oct 30 18:50:41.729932 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.729654    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb" podUID="af1375ab-417e-4a88-880e-f3f7a171d63e" containerName="payload-processing" containerID="cri-o://56ff4b9ae548218030f58ce47b20fcb0e9e9fd2084a0fe81eb7f55ddeeaa5de3" gracePeriod=25
Oct 30 18:50:41.743803 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.743756    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:50:41.816773 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.816745    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcdfd538-a850-48e9-883f-f7428503b8a9-kube-api-access-vd9dn\" (UniqueName: \"kubernetes.io/projected/dcdfd538-a850-48e9-883f-f7428503b8a9-kube-api-access-vd9dn\") pod \"dcdfd538-a850-48e9-883f-f7428503b8a9\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") "
Oct 30 18:50:41.816962 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.816850    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-workload-socket\") pod \"dcdfd538-a850-48e9-883f-f7428503b8a9\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") "
Oct 30 18:50:41.816962 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.816883    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-podinfo\") pod \"dcdfd538-a850-48e9-883f-f7428503b8a9\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") "
Oct 30 18:50:41.817092 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.816992    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-workload-certs\") pod \"dcdfd538-a850-48e9-883f-f7428503b8a9\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") "
Oct 30 18:50:41.817092 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.817075    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-envoy\") pod \"dcdfd538-a850-48e9-883f-f7428503b8a9\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") "
Oct 30 18:50:41.817259 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.817170    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcdfd538-a850-48e9-883f-f7428503b8a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdfd538-a850-48e9-883f-f7428503b8a9-istiod-ca-cert\") pod \"dcdfd538-a850-48e9-883f-f7428503b8a9\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") "
Oct 30 18:50:41.817259 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.817219    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-credential-socket\") pod \"dcdfd538-a850-48e9-883f-f7428503b8a9\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") "
Oct 30 18:50:41.817372 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.817270    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-token\" (UniqueName: \"kubernetes.io/projected/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-token\") pod \"dcdfd538-a850-48e9-883f-f7428503b8a9\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") "
Oct 30 18:50:41.817372 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.817306    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-data\") pod \"dcdfd538-a850-48e9-883f-f7428503b8a9\" (UID: \"dcdfd538-a850-48e9-883f-f7428503b8a9\") "
Oct 30 18:50:41.819104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.818093    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-credential-socket" pod "dcdfd538-a850-48e9-883f-f7428503b8a9" (UID: "dcdfd538-a850-48e9-883f-f7428503b8a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:41.819104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.818101    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcdfd538-a850-48e9-883f-f7428503b8a9-istiod-ca-cert" pod "dcdfd538-a850-48e9-883f-f7428503b8a9" (UID: "dcdfd538-a850-48e9-883f-f7428503b8a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:41.819104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.818108    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-workload-certs" pod "dcdfd538-a850-48e9-883f-f7428503b8a9" (UID: "dcdfd538-a850-48e9-883f-f7428503b8a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:41.819104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.818336    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-data" pod "dcdfd538-a850-48e9-883f-f7428503b8a9" (UID: "dcdfd538-a850-48e9-883f-f7428503b8a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:41.819104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.818455    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-workload-socket" pod "dcdfd538-a850-48e9-883f-f7428503b8a9" (UID: "dcdfd538-a850-48e9-883f-f7428503b8a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:41.819104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.818961    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:41.819104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.818983    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:41.819104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.818999    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdfd538-a850-48e9-883f-f7428503b8a9-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:41.819104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.819014    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:41.819104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.819028    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:41.819894 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.819868    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdfd538-a850-48e9-883f-f7428503b8a9-kube-api-access-vd9dn" pod "dcdfd538-a850-48e9-883f-f7428503b8a9" (UID: "dcdfd538-a850-48e9-883f-f7428503b8a9"). InnerVolumeSpecName "kube-api-access-vd9dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:41.821474 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.821432    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-envoy" pod "dcdfd538-a850-48e9-883f-f7428503b8a9" (UID: "dcdfd538-a850-48e9-883f-f7428503b8a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:41.821613 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.821584    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-podinfo" pod "dcdfd538-a850-48e9-883f-f7428503b8a9" (UID: "dcdfd538-a850-48e9-883f-f7428503b8a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:50:41.822719 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.822691    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-token" pod "dcdfd538-a850-48e9-883f-f7428503b8a9" (UID: "dcdfd538-a850-48e9-883f-f7428503b8a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:41.920558 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.920521    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:41.920558 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.920554    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:41.920788 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.920569    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd9dn\" (UniqueName: \"kubernetes.io/projected/dcdfd538-a850-48e9-883f-f7428503b8a9-kube-api-access-vd9dn\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:41.920788 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:41.920584    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdfd538-a850-48e9-883f-f7428503b8a9-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.053492 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.053466    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"
Oct 30 18:50:42.123158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.123066    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1375ab-417e-4a88-880e-f3f7a171d63e-kube-api-access-f48tt\" (UniqueName: \"kubernetes.io/projected/af1375ab-417e-4a88-880e-f3f7a171d63e-kube-api-access-f48tt\") pod \"af1375ab-417e-4a88-880e-f3f7a171d63e\" (UID: \"af1375ab-417e-4a88-880e-f3f7a171d63e\") "
Oct 30 18:50:42.123158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.123099    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1375ab-417e-4a88-880e-f3f7a171d63e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1375ab-417e-4a88-880e-f3f7a171d63e-plugins-config-volume\") pod \"af1375ab-417e-4a88-880e-f3f7a171d63e\" (UID: \"af1375ab-417e-4a88-880e-f3f7a171d63e\") "
Oct 30 18:50:42.123631 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.123602    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1375ab-417e-4a88-880e-f3f7a171d63e-plugins-config-volume" pod "af1375ab-417e-4a88-880e-f3f7a171d63e" (UID: "af1375ab-417e-4a88-880e-f3f7a171d63e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:42.123963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.123942    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1375ab-417e-4a88-880e-f3f7a171d63e-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.126003 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.125974    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1375ab-417e-4a88-880e-f3f7a171d63e-kube-api-access-f48tt" pod "af1375ab-417e-4a88-880e-f3f7a171d63e" (UID: "af1375ab-417e-4a88-880e-f3f7a171d63e"). InnerVolumeSpecName "kube-api-access-f48tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:42.224945 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.224913    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f48tt\" (UniqueName: \"kubernetes.io/projected/af1375ab-417e-4a88-880e-f3f7a171d63e-kube-api-access-f48tt\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:42.569641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.569553    4372 generic.go:356] "Generic (PLEG): container finished" podID="af1375ab-417e-4a88-880e-f3f7a171d63e" containerID="56ff4b9ae548218030f58ce47b20fcb0e9e9fd2084a0fe81eb7f55ddeeaa5de3" exitCode=0
Oct 30 18:50:42.569641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.569602    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"
Oct 30 18:50:42.570180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.569641    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb" event={"ID":"af1375ab-417e-4a88-880e-f3f7a171d63e","Type":"ContainerDied","Data":"56ff4b9ae548218030f58ce47b20fcb0e9e9fd2084a0fe81eb7f55ddeeaa5de3"}
Oct 30 18:50:42.570180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.569708    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb" event={"ID":"af1375ab-417e-4a88-880e-f3f7a171d63e","Type":"ContainerDied","Data":"f618fc760b7d35a27688f3d3f6fa61f8e12129f1461f027365d49f2be80b6988"}
Oct 30 18:50:42.570180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.569742    4372 scope.go:122] "RemoveContainer" containerID="56ff4b9ae548218030f58ce47b20fcb0e9e9fd2084a0fe81eb7f55ddeeaa5de3"
Oct 30 18:50:42.572185 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.572158    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn" event={"ID":"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3","Type":"ContainerStarted","Data":"f5ec8dcbcf230ce2f42280cb2a1177793e9d50a4930723dc0786336aab05d7ec"}
Oct 30 18:50:42.573648 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.573633    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"
Oct 30 18:50:42.573760 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.573649    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69" event={"ID":"dcdfd538-a850-48e9-883f-f7428503b8a9","Type":"ContainerDied","Data":"7d698369a2266df5ed0f1086cf45af38613376f391ec956fcbb2b2f5099c89a5"}
Oct 30 18:50:42.581535 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.581521    4372 scope.go:122] "RemoveContainer" containerID="56ff4b9ae548218030f58ce47b20fcb0e9e9fd2084a0fe81eb7f55ddeeaa5de3"
Oct 30 18:50:42.581809 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:50:42.581789    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ff4b9ae548218030f58ce47b20fcb0e9e9fd2084a0fe81eb7f55ddeeaa5de3\": container with ID starting with 56ff4b9ae548218030f58ce47b20fcb0e9e9fd2084a0fe81eb7f55ddeeaa5de3 not found: ID does not exist" containerID="56ff4b9ae548218030f58ce47b20fcb0e9e9fd2084a0fe81eb7f55ddeeaa5de3"
Oct 30 18:50:42.581856 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.581817    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ff4b9ae548218030f58ce47b20fcb0e9e9fd2084a0fe81eb7f55ddeeaa5de3"} err="failed to get container status \"56ff4b9ae548218030f58ce47b20fcb0e9e9fd2084a0fe81eb7f55ddeeaa5de3\": rpc error: code = NotFound desc = could not find container \"56ff4b9ae548218030f58ce47b20fcb0e9e9fd2084a0fe81eb7f55ddeeaa5de3\": container with ID starting with 56ff4b9ae548218030f58ce47b20fcb0e9e9fd2084a0fe81eb7f55ddeeaa5de3 not found: ID does not exist"
Oct 30 18:50:42.581856 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.581833    4372 scope.go:122] "RemoveContainer" containerID="7451ab21d620e6bf75ec88c0997ab82aeaf30f64bea334f9e2a2380b8bd9a87c"
Oct 30 18:50:42.594943 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.594919    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"]
Oct 30 18:50:42.598520 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.598496    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9521e-5d6b95698d-z92sb"]
Oct 30 18:50:42.648397 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.648373    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"]
Oct 30 18:50:42.653828 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:42.653803    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-67k69"]
Oct 30 18:50:43.421601 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:43.421566    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:50:43.421952 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:43.421929    4372 patch_prober.go:29] interesting pod/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 18:50:43.422166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:43.422102    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn" podUID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 18:50:44.307151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:44.299951    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1375ab-417e-4a88-880e-f3f7a171d63e" path="/var/lib/kubelet/pods/af1375ab-417e-4a88-880e-f3f7a171d63e/volumes"
Oct 30 18:50:44.307151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:44.301046    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcdfd538-a850-48e9-883f-f7428503b8a9" path="/var/lib/kubelet/pods/dcdfd538-a850-48e9-883f-f7428503b8a9/volumes"
Oct 30 18:50:44.421580 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:44.421546    4372 patch_prober.go:29] interesting pod/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 18:50:44.421777 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:44.421598    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn" podUID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 18:50:44.816826 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:44.816791    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"]
Oct 30 18:50:44.817028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:44.817004    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5" podUID="2a668eb1-f2a4-4819-8d77-2b82b4884a8d" containerName="maas-api" containerID="cri-o://7c474fec40d0084a7087ef5469f85c3c46ca671c40b01b43e58a3d46fac89a5b" gracePeriod=30
Oct 30 18:50:45.189612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.189312    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:50:45.271926 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.271887    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-metrics-certs\") pod \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") "
Oct 30 18:50:45.272114 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.271967    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-maas-api-tls\") pod \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") "
Oct 30 18:50:45.272114 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.272035    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-openshift-service-ca\") pod \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") "
Oct 30 18:50:45.272114 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.272073    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-kube-api-access-fpvgb\" (UniqueName: \"kubernetes.io/projected/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-kube-api-access-fpvgb\") pod \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\" (UID: \"2a668eb1-f2a4-4819-8d77-2b82b4884a8d\") "
Oct 30 18:50:45.272817 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.272786    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-openshift-service-ca" pod "2a668eb1-f2a4-4819-8d77-2b82b4884a8d" (UID: "2a668eb1-f2a4-4819-8d77-2b82b4884a8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:45.274594 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.274564    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-kube-api-access-fpvgb" pod "2a668eb1-f2a4-4819-8d77-2b82b4884a8d" (UID: "2a668eb1-f2a4-4819-8d77-2b82b4884a8d"). InnerVolumeSpecName "kube-api-access-fpvgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:45.274905 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.274874    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-maas-api-tls" pod "2a668eb1-f2a4-4819-8d77-2b82b4884a8d" (UID: "2a668eb1-f2a4-4819-8d77-2b82b4884a8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:50:45.275148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.275090    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-metrics-certs" pod "2a668eb1-f2a4-4819-8d77-2b82b4884a8d" (UID: "2a668eb1-f2a4-4819-8d77-2b82b4884a8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:50:45.408197 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.407903    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:45.408197 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.407939    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:45.408197 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.407956    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:45.408197 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.407971    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpvgb\" (UniqueName: \"kubernetes.io/projected/2a668eb1-f2a4-4819-8d77-2b82b4884a8d-kube-api-access-fpvgb\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:45.440603 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.440528    4372 patch_prober.go:29] interesting pod/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 18:50:45.440801 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.440629    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn" podUID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 18:50:45.754283 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.754200    4372 generic.go:356] "Generic (PLEG): container finished" podID="2a668eb1-f2a4-4819-8d77-2b82b4884a8d" containerID="7c474fec40d0084a7087ef5469f85c3c46ca671c40b01b43e58a3d46fac89a5b" exitCode=0
Oct 30 18:50:45.754505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.754351    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"
Oct 30 18:50:45.754796 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.754385    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5" event={"ID":"2a668eb1-f2a4-4819-8d77-2b82b4884a8d","Type":"ContainerDied","Data":"7c474fec40d0084a7087ef5469f85c3c46ca671c40b01b43e58a3d46fac89a5b"}
Oct 30 18:50:45.754901 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.754880    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5" event={"ID":"2a668eb1-f2a4-4819-8d77-2b82b4884a8d","Type":"ContainerDied","Data":"3707ed6d8de14f30d4ff37b9ba3a1519aedcef234bc819979e1d508dc3a94553"}
Oct 30 18:50:45.754972 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.754924    4372 scope.go:122] "RemoveContainer" containerID="7c474fec40d0084a7087ef5469f85c3c46ca671c40b01b43e58a3d46fac89a5b"
Oct 30 18:50:45.766891 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.766870    4372 scope.go:122] "RemoveContainer" containerID="7c474fec40d0084a7087ef5469f85c3c46ca671c40b01b43e58a3d46fac89a5b"
Oct 30 18:50:45.774743 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:50:45.774464    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c474fec40d0084a7087ef5469f85c3c46ca671c40b01b43e58a3d46fac89a5b\": container with ID starting with 7c474fec40d0084a7087ef5469f85c3c46ca671c40b01b43e58a3d46fac89a5b not found: ID does not exist" containerID="7c474fec40d0084a7087ef5469f85c3c46ca671c40b01b43e58a3d46fac89a5b"
Oct 30 18:50:45.775239 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.775200    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c474fec40d0084a7087ef5469f85c3c46ca671c40b01b43e58a3d46fac89a5b"} err="failed to get container status \"7c474fec40d0084a7087ef5469f85c3c46ca671c40b01b43e58a3d46fac89a5b\": rpc error: code = NotFound desc = could not find container \"7c474fec40d0084a7087ef5469f85c3c46ca671c40b01b43e58a3d46fac89a5b\": container with ID starting with 7c474fec40d0084a7087ef5469f85c3c46ca671c40b01b43e58a3d46fac89a5b not found: ID does not exist"
Oct 30 18:50:45.877883 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.877843    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"]
Oct 30 18:50:45.888153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:45.886338    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30970e-5f5fcffdc7-55jr5"]
Oct 30 18:50:46.268324 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:46.268296    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a668eb1-f2a4-4819-8d77-2b82b4884a8d" path="/var/lib/kubelet/pods/2a668eb1-f2a4-4819-8d77-2b82b4884a8d/volumes"
Oct 30 18:50:46.421088 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:46.421056    4372 patch_prober.go:29] interesting pod/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 18:50:46.421498 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:46.421103    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn" podUID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 18:50:47.420651 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:47.420609    4372 patch_prober.go:29] interesting pod/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 18:50:47.420822 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:47.420695    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn" podUID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 18:50:48.455149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:48.454337    4372 patch_prober.go:29] interesting pod/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 18:50:48.455149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:48.454400    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn" podUID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 18:50:49.425572 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:49.425543    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:50:49.426062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:49.426039    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:50:49.426736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:49.426719    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:50:50.155142 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.155093    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"]
Oct 30 18:50:50.155907 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.155869    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x" podUID="6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6" containerName="maas-api" containerID="cri-o://9a09359aedff21a90a4ecc785ef70896a1c61abdff34409a546e38191494e5cb" gracePeriod=30
Oct 30 18:50:50.612626 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.612287    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:50:50.728789 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.726613    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-kube-api-access-rt588\" (UniqueName: \"kubernetes.io/projected/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-kube-api-access-rt588\") pod \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") "
Oct 30 18:50:50.728789 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.726797    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-openshift-service-ca\") pod \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") "
Oct 30 18:50:50.728789 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.726858    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-metrics-certs\") pod \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") "
Oct 30 18:50:50.728789 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.726902    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-maas-api-tls\") pod \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\" (UID: \"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6\") "
Oct 30 18:50:50.737665 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.737631    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-openshift-service-ca" pod "6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6" (UID: "6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:50.758669 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.758607    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-maas-api-tls" pod "6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6" (UID: "6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:50:50.790015 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.781116    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-kube-api-access-rt588" pod "6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6" (UID: "6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6"). InnerVolumeSpecName "kube-api-access-rt588". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:50.790015 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.782618    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-metrics-certs" pod "6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6" (UID: "6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:50:50.831292 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.831258    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt588\" (UniqueName: \"kubernetes.io/projected/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-kube-api-access-rt588\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:50.831292 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.831288    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:50.831292 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.831298    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:50.831523 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.831307    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:50.864109 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.864078    4372 generic.go:356] "Generic (PLEG): container finished" podID="6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6" containerID="9a09359aedff21a90a4ecc785ef70896a1c61abdff34409a546e38191494e5cb" exitCode=0
Oct 30 18:50:50.864333 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.864166    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"
Oct 30 18:50:50.864333 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.864186    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x" event={"ID":"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6","Type":"ContainerDied","Data":"9a09359aedff21a90a4ecc785ef70896a1c61abdff34409a546e38191494e5cb"}
Oct 30 18:50:50.864333 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.864243    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x" event={"ID":"6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6","Type":"ContainerDied","Data":"e855487da9cacd91e8569be8a3e3b9fe3674e6859db06000f169f7a05b7497e7"}
Oct 30 18:50:50.864333 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.864282    4372 scope.go:122] "RemoveContainer" containerID="9a09359aedff21a90a4ecc785ef70896a1c61abdff34409a546e38191494e5cb"
Oct 30 18:50:50.874977 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.874962    4372 scope.go:122] "RemoveContainer" containerID="9a09359aedff21a90a4ecc785ef70896a1c61abdff34409a546e38191494e5cb"
Oct 30 18:50:50.875292 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:50:50.875273    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a09359aedff21a90a4ecc785ef70896a1c61abdff34409a546e38191494e5cb\": container with ID starting with 9a09359aedff21a90a4ecc785ef70896a1c61abdff34409a546e38191494e5cb not found: ID does not exist" containerID="9a09359aedff21a90a4ecc785ef70896a1c61abdff34409a546e38191494e5cb"
Oct 30 18:50:50.875372 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.875301    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a09359aedff21a90a4ecc785ef70896a1c61abdff34409a546e38191494e5cb"} err="failed to get container status \"9a09359aedff21a90a4ecc785ef70896a1c61abdff34409a546e38191494e5cb\": rpc error: code = NotFound desc = could not find container \"9a09359aedff21a90a4ecc785ef70896a1c61abdff34409a546e38191494e5cb\": container with ID starting with 9a09359aedff21a90a4ecc785ef70896a1c61abdff34409a546e38191494e5cb not found: ID does not exist"
Oct 30 18:50:50.942215 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.942195    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"]
Oct 30 18:50:50.948502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:50.948468    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9521e-df56d685-gbh6x"]
Oct 30 18:50:52.265929 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:52.265898    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6" path="/var/lib/kubelet/pods/6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6/volumes"
Oct 30 18:50:55.196202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:55.196171    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"]
Oct 30 18:50:55.196596 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:55.196396    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn" podUID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" containerName="istio-proxy" containerID="cri-o://f5ec8dcbcf230ce2f42280cb2a1177793e9d50a4930723dc0786336aab05d7ec" gracePeriod=30
Oct 30 18:50:57.696206 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:50:57.696165    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:50:59.098948 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.098916    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"]
Oct 30 18:50:59.099473 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099459    4372 cpu_manager.go:414] "containerMap: removing container" podUID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" containerName="istio-proxy"
Oct 30 18:50:59.099533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099473    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" containerName="istio-proxy"
Oct 30 18:50:59.099533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099491    4372 cpu_manager.go:414] "containerMap: removing container" podUID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" containerName="istio-proxy"
Oct 30 18:50:59.099533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099496    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" containerName="istio-proxy"
Oct 30 18:50:59.099533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099507    4372 cpu_manager.go:414] "containerMap: removing container" podUID="01404f15-9f9f-446c-88d0-83e2f67bd877" containerName="payload-processing"
Oct 30 18:50:59.099533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099512    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01404f15-9f9f-446c-88d0-83e2f67bd877" containerName="payload-processing"
Oct 30 18:50:59.099533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099529    4372 cpu_manager.go:414] "containerMap: removing container" podUID="2a668eb1-f2a4-4819-8d77-2b82b4884a8d" containerName="maas-api"
Oct 30 18:50:59.099533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099533    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a668eb1-f2a4-4819-8d77-2b82b4884a8d" containerName="maas-api"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099541    4372 cpu_manager.go:414] "containerMap: removing container" podUID="dcdfd538-a850-48e9-883f-f7428503b8a9" containerName="istio-proxy"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099546    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcdfd538-a850-48e9-883f-f7428503b8a9" containerName="istio-proxy"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099556    4372 cpu_manager.go:414] "containerMap: removing container" podUID="82c34738-0746-4ea1-840e-3ab7514980c9" containerName="test-curl-7411-6148bf"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099560    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82c34738-0746-4ea1-840e-3ab7514980c9" containerName="test-curl-7411-6148bf"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099570    4372 cpu_manager.go:414] "containerMap: removing container" podUID="8745c459-008e-471d-a4b4-67da58c8432d" containerName="maas-api"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099574    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8745c459-008e-471d-a4b4-67da58c8432d" containerName="maas-api"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099580    4372 cpu_manager.go:414] "containerMap: removing container" podUID="af1375ab-417e-4a88-880e-f3f7a171d63e" containerName="payload-processing"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099584    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1375ab-417e-4a88-880e-f3f7a171d63e" containerName="payload-processing"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099591    4372 cpu_manager.go:414] "containerMap: removing container" podUID="6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6" containerName="maas-api"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099595    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6" containerName="maas-api"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099673    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a668eb1-f2a4-4819-8d77-2b82b4884a8d" containerName="maas-api"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099679    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" containerName="istio-proxy"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099684    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcdfd538-a850-48e9-883f-f7428503b8a9" containerName="istio-proxy"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099691    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="82c34738-0746-4ea1-840e-3ab7514980c9" containerName="test-curl-7411-6148bf"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099698    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="8745c459-008e-471d-a4b4-67da58c8432d" containerName="maas-api"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099703    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="01404f15-9f9f-446c-88d0-83e2f67bd877" containerName="payload-processing"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099709    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1375ab-417e-4a88-880e-f3f7a171d63e" containerName="payload-processing"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099714    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bdafb2d-1a2f-422b-87c7-d29c85e4b7b6" containerName="maas-api"
Oct 30 18:50:59.099783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.099719    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca9ec230-0cb3-4296-a9f1-58d09a8e17b4" containerName="istio-proxy"
Oct 30 18:50:59.103155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.103138    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"
Oct 30 18:50:59.108809 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.108789    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2aada387\""
Oct 30 18:50:59.117838 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.117818    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2aada387-dockercfg-6dll4\""
Oct 30 18:50:59.119017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.118991    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"]
Oct 30 18:50:59.124911 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.124891    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"]
Oct 30 18:50:59.128841 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.128818    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"
Oct 30 18:50:59.146065 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.146041    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"]
Oct 30 18:50:59.208147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.200643    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ltj\" (UniqueName: \"kubernetes.io/projected/494f7ef0-35bc-4c22-85d0-8799b860298e-kube-api-access-l2ltj\") pod \"payload-processing-e2e-mt-2aada387-6db6c946b-89r8w\" (UID: \"494f7ef0-35bc-4c22-85d0-8799b860298e\") " pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"
Oct 30 18:50:59.208147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.200754    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494f7ef0-35bc-4c22-85d0-8799b860298e-plugins-config-volume\") pod \"payload-processing-e2e-mt-2aada387-6db6c946b-89r8w\" (UID: \"494f7ef0-35bc-4c22-85d0-8799b860298e\") " pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"
Oct 30 18:50:59.208147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.200893    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73af2591-d3cd-42ff-a60a-dfa5107562ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn\" (UID: \"73af2591-d3cd-42ff-a60a-dfa5107562ed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"
Oct 30 18:50:59.208147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.200932    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs6w5\" (UniqueName: \"kubernetes.io/projected/73af2591-d3cd-42ff-a60a-dfa5107562ed-kube-api-access-hs6w5\") pod \"payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn\" (UID: \"73af2591-d3cd-42ff-a60a-dfa5107562ed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"
Oct 30 18:50:59.303152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.302240    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ltj\" (UniqueName: \"kubernetes.io/projected/494f7ef0-35bc-4c22-85d0-8799b860298e-kube-api-access-l2ltj\") pod \"payload-processing-e2e-mt-2aada387-6db6c946b-89r8w\" (UID: \"494f7ef0-35bc-4c22-85d0-8799b860298e\") " pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"
Oct 30 18:50:59.303152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.302352    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494f7ef0-35bc-4c22-85d0-8799b860298e-plugins-config-volume\") pod \"payload-processing-e2e-mt-2aada387-6db6c946b-89r8w\" (UID: \"494f7ef0-35bc-4c22-85d0-8799b860298e\") " pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"
Oct 30 18:50:59.303152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.302516    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73af2591-d3cd-42ff-a60a-dfa5107562ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn\" (UID: \"73af2591-d3cd-42ff-a60a-dfa5107562ed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"
Oct 30 18:50:59.303152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.302558    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs6w5\" (UniqueName: \"kubernetes.io/projected/73af2591-d3cd-42ff-a60a-dfa5107562ed-kube-api-access-hs6w5\") pod \"payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn\" (UID: \"73af2591-d3cd-42ff-a60a-dfa5107562ed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"
Oct 30 18:50:59.307151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.304927    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494f7ef0-35bc-4c22-85d0-8799b860298e-plugins-config-volume\") pod \"payload-processing-e2e-mt-2aada387-6db6c946b-89r8w\" (UID: \"494f7ef0-35bc-4c22-85d0-8799b860298e\") " pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"
Oct 30 18:50:59.343150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.332071    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73af2591-d3cd-42ff-a60a-dfa5107562ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn\" (UID: \"73af2591-d3cd-42ff-a60a-dfa5107562ed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"
Oct 30 18:50:59.350819 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.349951    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ltj\" (UniqueName: \"kubernetes.io/projected/494f7ef0-35bc-4c22-85d0-8799b860298e-kube-api-access-l2ltj\") pod \"payload-processing-e2e-mt-2aada387-6db6c946b-89r8w\" (UID: \"494f7ef0-35bc-4c22-85d0-8799b860298e\") " pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"
Oct 30 18:50:59.401149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.400896    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs6w5\" (UniqueName: \"kubernetes.io/projected/73af2591-d3cd-42ff-a60a-dfa5107562ed-kube-api-access-hs6w5\") pod \"payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn\" (UID: \"73af2591-d3cd-42ff-a60a-dfa5107562ed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"
Oct 30 18:50:59.431411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.431381    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"
Oct 30 18:50:59.448542 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.448499    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"
Oct 30 18:50:59.513858 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.513783    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:50:59.965504 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.965473    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"]
Oct 30 18:50:59.966592 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:50:59.966561    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73af2591_d3cd_42ff_a60a_dfa5107562ed.slice/crio-ada7ceac1d770ab884ecf4f92a661571fc15e959432c430cb170961275520081 WatchSource:0}: Error finding container ada7ceac1d770ab884ecf4f92a661571fc15e959432c430cb170961275520081: Status 404 returned error can't find the container with id ada7ceac1d770ab884ecf4f92a661571fc15e959432c430cb170961275520081
Oct 30 18:50:59.983660 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:50:59.983639    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"]
Oct 30 18:50:59.984538 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:50:59.984504    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494f7ef0_35bc_4c22_85d0_8799b860298e.slice/crio-600a78147031ad515628662fcd910f46e1ff7f6fcc3028fb0acffbf99005e96b WatchSource:0}: Error finding container 600a78147031ad515628662fcd910f46e1ff7f6fcc3028fb0acffbf99005e96b: Status 404 returned error can't find the container with id 600a78147031ad515628662fcd910f46e1ff7f6fcc3028fb0acffbf99005e96b
Oct 30 18:51:00.846264 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.846241    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:51:00.948087 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.948046    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w" event={"ID":"494f7ef0-35bc-4c22-85d0-8799b860298e","Type":"ContainerStarted","Data":"24e9c6d6e7d326434c0b5e368c5ed14f6c7c1440140ec839f3b8a48e7813a4cc"}
Oct 30 18:51:00.948279 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.948173    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w" event={"ID":"494f7ef0-35bc-4c22-85d0-8799b860298e","Type":"ContainerStarted","Data":"600a78147031ad515628662fcd910f46e1ff7f6fcc3028fb0acffbf99005e96b"}
Oct 30 18:51:00.950061 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.950019    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn" event={"ID":"73af2591-d3cd-42ff-a60a-dfa5107562ed","Type":"ContainerStarted","Data":"ebf8c6f2ae52679a0980b5b12ec2c6f7e9bf34e1b5880720368de2bd58154260"}
Oct 30 18:51:00.950218 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.950090    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn" event={"ID":"73af2591-d3cd-42ff-a60a-dfa5107562ed","Type":"ContainerStarted","Data":"ada7ceac1d770ab884ecf4f92a661571fc15e959432c430cb170961275520081"}
Oct 30 18:51:00.950218 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.950174    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"
Oct 30 18:51:00.955207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.951861    4372 generic.go:356] "Generic (PLEG): container finished" podID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" containerID="f5ec8dcbcf230ce2f42280cb2a1177793e9d50a4930723dc0786336aab05d7ec" exitCode=0
Oct 30 18:51:00.955207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.951891    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn" event={"ID":"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3","Type":"ContainerDied","Data":"f5ec8dcbcf230ce2f42280cb2a1177793e9d50a4930723dc0786336aab05d7ec"}
Oct 30 18:51:00.955207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.951925    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn" event={"ID":"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3","Type":"ContainerDied","Data":"1ff303353df1ba60f3693986102267d088c7e8a73a4f149ad12872ded0750b23"}
Oct 30 18:51:00.955207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.951947    4372 scope.go:122] "RemoveContainer" containerID="f5ec8dcbcf230ce2f42280cb2a1177793e9d50a4930723dc0786336aab05d7ec"
Oct 30 18:51:00.955207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.952083    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"
Oct 30 18:51:00.971997 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.971974    4372 scope.go:122] "RemoveContainer" containerID="665eeabdf2bc826c6ee54928b394063be812396e1cc4002b419620c08d44fb39"
Oct 30 18:51:00.977110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.977069    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w" podStartSLOduration=1.9770569 podStartE2EDuration="1.9770569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:00.970658468 +0000 UTC m=+1074.839076800" watchObservedRunningTime="2026-10-30 18:51:00.9770569 +0000 UTC m=+1074.845475230"
Oct 30 18:51:00.985822 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.985800    4372 scope.go:122] "RemoveContainer" containerID="f5ec8dcbcf230ce2f42280cb2a1177793e9d50a4930723dc0786336aab05d7ec"
Oct 30 18:51:00.986229 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:51:00.986195    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ec8dcbcf230ce2f42280cb2a1177793e9d50a4930723dc0786336aab05d7ec\": container with ID starting with f5ec8dcbcf230ce2f42280cb2a1177793e9d50a4930723dc0786336aab05d7ec not found: ID does not exist" containerID="f5ec8dcbcf230ce2f42280cb2a1177793e9d50a4930723dc0786336aab05d7ec"
Oct 30 18:51:00.986328 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.986236    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ec8dcbcf230ce2f42280cb2a1177793e9d50a4930723dc0786336aab05d7ec"} err="failed to get container status \"f5ec8dcbcf230ce2f42280cb2a1177793e9d50a4930723dc0786336aab05d7ec\": rpc error: code = NotFound desc = could not find container \"f5ec8dcbcf230ce2f42280cb2a1177793e9d50a4930723dc0786336aab05d7ec\": container with ID starting with f5ec8dcbcf230ce2f42280cb2a1177793e9d50a4930723dc0786336aab05d7ec not found: ID does not exist"
Oct 30 18:51:00.986328 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.986261    4372 scope.go:122] "RemoveContainer" containerID="665eeabdf2bc826c6ee54928b394063be812396e1cc4002b419620c08d44fb39"
Oct 30 18:51:00.986581 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:51:00.986556    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665eeabdf2bc826c6ee54928b394063be812396e1cc4002b419620c08d44fb39\": container with ID starting with 665eeabdf2bc826c6ee54928b394063be812396e1cc4002b419620c08d44fb39 not found: ID does not exist" containerID="665eeabdf2bc826c6ee54928b394063be812396e1cc4002b419620c08d44fb39"
Oct 30 18:51:00.986698 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.986588    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665eeabdf2bc826c6ee54928b394063be812396e1cc4002b419620c08d44fb39"} err="failed to get container status \"665eeabdf2bc826c6ee54928b394063be812396e1cc4002b419620c08d44fb39\": rpc error: code = NotFound desc = could not find container \"665eeabdf2bc826c6ee54928b394063be812396e1cc4002b419620c08d44fb39\": container with ID starting with 665eeabdf2bc826c6ee54928b394063be812396e1cc4002b419620c08d44fb39 not found: ID does not exist"
Oct 30 18:51:00.999001 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.998978    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-workload-socket\") pod \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") "
Oct 30 18:51:00.999101 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.999030    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-podinfo\") pod \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") "
Oct 30 18:51:00.999181 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.999137    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-kube-api-access-qmnhf\" (UniqueName: \"kubernetes.io/projected/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-kube-api-access-qmnhf\") pod \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") "
Oct 30 18:51:00.999284 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.999262    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-workload-socket" pod "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" (UID: "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:00.999636 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.999612    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-workload-certs\") pod \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") "
Oct 30 18:51:00.999734 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.999657    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-credential-socket\") pod \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") "
Oct 30 18:51:00.999734 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.999688    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-envoy\") pod \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") "
Oct 30 18:51:00.999841 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.999757    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-token\" (UniqueName: \"kubernetes.io/projected/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-token\") pod \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") "
Oct 30 18:51:00.999841 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.999801    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istiod-ca-cert\") pod \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") "
Oct 30 18:51:00.999988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.999847    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-data\") pod \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\" (UID: \"d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3\") "
Oct 30 18:51:01.000042 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.999980    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-credential-socket" pod "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" (UID: "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:01.000042 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:00.999994    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-workload-certs" pod "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" (UID: "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:01.000722 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.000430    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-data" pod "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" (UID: "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:01.000722 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.000451    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istiod-ca-cert" pod "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" (UID: "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:01.000722 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.000603    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.000722 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.000623    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.000722 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.000638    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.000722 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.000651    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.000722 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.000664    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.001812 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.001791    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-kube-api-access-qmnhf" pod "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" (UID: "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3"). InnerVolumeSpecName "kube-api-access-qmnhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:01.002052 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.002025    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-envoy" pod "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" (UID: "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:01.002175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.002107    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-podinfo" pod "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" (UID: "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:51:01.002566 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.002542    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-token" pod "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" (UID: "d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:01.013118 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.013073    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn" podStartSLOduration=2.013058331 podStartE2EDuration="2.013058331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:01.009617646 +0000 UTC m=+1074.878035975" watchObservedRunningTime="2026-10-30 18:51:01.013058331 +0000 UTC m=+1074.881476662"
Oct 30 18:51:01.101671 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.101639    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.101776 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.101675    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmnhf\" (UniqueName: \"kubernetes.io/projected/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-kube-api-access-qmnhf\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.101776 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.101689    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.101776 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.101702    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.232442 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.232414    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"]
Oct 30 18:51:01.232970 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.232958    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" containerName="istio-proxy"
Oct 30 18:51:01.233025 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.232972    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" containerName="istio-proxy"
Oct 30 18:51:01.233088 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.233080    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" containerName="istio-proxy"
Oct 30 18:51:01.233120 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.233090    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" containerName="istio-proxy"
Oct 30 18:51:01.237874 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.237853    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.243331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.243302    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-52fa8b-openshift-default-dockercfg-vdwmp\""
Oct 30 18:51:01.275230 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.275200    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"]
Oct 30 18:51:01.306515 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.306478    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-podinfo\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.306678 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.306563    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-token\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.306678 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.306599    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-workload-socket\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.306678 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.306669    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-envoy\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.306876 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.306729    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-credential-socket\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.306876 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.306790    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-workload-certs\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.306981 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.306895    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxrdx\" (UniqueName: \"kubernetes.io/projected/0328c404-96a0-447a-b06c-a1ef68ba3178-kube-api-access-fxrdx\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.306981 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.306948    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0328c404-96a0-447a-b06c-a1ef68ba3178-istiod-ca-cert\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.307114 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.307039    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-data\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.310423 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.310392    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"]
Oct 30 18:51:01.316103 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.316078    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-30970e-openshift-default-d96c4969f-drgkn"]
Oct 30 18:51:01.408475 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.408416    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxrdx\" (UniqueName: \"kubernetes.io/projected/0328c404-96a0-447a-b06c-a1ef68ba3178-kube-api-access-fxrdx\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.408475 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.408456    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0328c404-96a0-447a-b06c-a1ef68ba3178-istiod-ca-cert\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.408644 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.408502    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-data\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.408705 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.408657    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-podinfo\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.408770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.408721    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-token\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.408770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.408764    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-workload-socket\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.408882 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.408808    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-envoy\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.408882 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.408848    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-credential-socket\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.408993 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.408889    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-workload-certs\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.408993 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.408847    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-data\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.409252 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.409229    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-workload-socket\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.409329 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.409251    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-credential-socket\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.409394 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.409327    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-workload-certs\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.409394 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.409351    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0328c404-96a0-447a-b06c-a1ef68ba3178-istiod-ca-cert\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.411156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.411116    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-envoy\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.411379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.411360    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-podinfo\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.425645 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.425619    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-token\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.425940 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.425919    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxrdx\" (UniqueName: \"kubernetes.io/projected/0328c404-96a0-447a-b06c-a1ef68ba3178-kube-api-access-fxrdx\") pod \"e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") " pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:01.553069 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:01.553042    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:02.013636 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:02.013547    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"
Oct 30 18:51:02.223930 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:02.223902    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"]
Oct 30 18:51:02.225042 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:51:02.225016    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0328c404_96a0_447a_b06c_a1ef68ba3178.slice/crio-944208a83081e87886991211c4deb47858abad820646f05e76a2750cad02d50a WatchSource:0}: Error finding container 944208a83081e87886991211c4deb47858abad820646f05e76a2750cad02d50a: Status 404 returned error can't find the container with id 944208a83081e87886991211c4deb47858abad820646f05e76a2750cad02d50a
Oct 30 18:51:02.227592 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:02.227559    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:02.227666 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:02.227628    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:02.227700 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:02.227668    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:02.266757 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:02.266735    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" path="/var/lib/kubelet/pods/d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3/volumes"
Oct 30 18:51:03.003444 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.003403    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"]
Oct 30 18:51:03.004003 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.003985    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" containerName="istio-proxy"
Oct 30 18:51:03.004003 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.004000    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d8a82d-1fd7-4d5b-a894-8efdb808b1c3" containerName="istio-proxy"
Oct 30 18:51:03.007717 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.007695    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.012259 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.012239    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5ad713-openshift-default-dockercfg-xfnkc\""
Oct 30 18:51:03.023556 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.023532    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"]
Oct 30 18:51:03.027219 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.027191    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t" event={"ID":"0328c404-96a0-447a-b06c-a1ef68ba3178","Type":"ContainerStarted","Data":"cdf726eec866bac5ec4c9315877bb58e09621cda2eca0f6c018ed444b0910714"}
Oct 30 18:51:03.027328 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.027249    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t" event={"ID":"0328c404-96a0-447a-b06c-a1ef68ba3178","Type":"ContainerStarted","Data":"944208a83081e87886991211c4deb47858abad820646f05e76a2750cad02d50a"}
Oct 30 18:51:03.052286 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.052254    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msmsz\" (UniqueName: \"kubernetes.io/projected/5a4347ef-492a-4569-a344-1e683419ea7f-kube-api-access-msmsz\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.052408 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.052367    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4347ef-492a-4569-a344-1e683419ea7f-istio-token\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.052488 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.052442    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-workload-socket\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.052593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.052577    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-credential-socket\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.052649 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.052606    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4347ef-492a-4569-a344-1e683419ea7f-istio-podinfo\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.052698 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.052662    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-istio-data\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.052752 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.052706    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-istio-envoy\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.052885 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.052866    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4347ef-492a-4569-a344-1e683419ea7f-istiod-ca-cert\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.052979 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.052960    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-workload-certs\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.086303 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.086255    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t" podStartSLOduration=2.08624328 podStartE2EDuration="2.08624328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:03.082951165 +0000 UTC m=+1076.951369482" watchObservedRunningTime="2026-10-30 18:51:03.08624328 +0000 UTC m=+1076.954661610"
Oct 30 18:51:03.154188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.154119    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-workload-certs\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.154324 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.154306    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msmsz\" (UniqueName: \"kubernetes.io/projected/5a4347ef-492a-4569-a344-1e683419ea7f-kube-api-access-msmsz\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.154381 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.154354    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4347ef-492a-4569-a344-1e683419ea7f-istio-token\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.154435 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.154403    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-workload-socket\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.154507 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.154472    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-workload-certs\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.154565 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.154493    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-credential-socket\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.154565 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.154532    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4347ef-492a-4569-a344-1e683419ea7f-istio-podinfo\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.154684 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.154580    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-istio-data\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.154684 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.154622    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-istio-envoy\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.154774 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.154681    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4347ef-492a-4569-a344-1e683419ea7f-istiod-ca-cert\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.154774 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.154735    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-workload-socket\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.154856 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.154783    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-credential-socket\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.154951 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.154930    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-istio-data\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.155289 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.155273    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4347ef-492a-4569-a344-1e683419ea7f-istiod-ca-cert\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.156977 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.156956    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-istio-envoy\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.157231 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.157211    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4347ef-492a-4569-a344-1e683419ea7f-istio-podinfo\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.167113 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.167091    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4347ef-492a-4569-a344-1e683419ea7f-istio-token\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.167533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.167510    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msmsz\" (UniqueName: \"kubernetes.io/projected/5a4347ef-492a-4569-a344-1e683419ea7f-kube-api-access-msmsz\") pod \"e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") " pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.337241 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.336407    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:03.555149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.554522    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:03.625226 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.621879    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:03.978533 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:51:03.978500    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4347ef_492a_4569_a344_1e683419ea7f.slice/crio-59218e52c6ab0fdf7e5b1808f2f997c35bcf9adddffb7c6b12a64a191e753b93 WatchSource:0}: Error finding container 59218e52c6ab0fdf7e5b1808f2f997c35bcf9adddffb7c6b12a64a191e753b93: Status 404 returned error can't find the container with id 59218e52c6ab0fdf7e5b1808f2f997c35bcf9adddffb7c6b12a64a191e753b93
Oct 30 18:51:03.978640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.978592    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"]
Oct 30 18:51:03.981215 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.981183    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:03.981322 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.981246    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:03.981322 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:03.981274    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:04.037934 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.037689    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h" event={"ID":"5a4347ef-492a-4569-a344-1e683419ea7f","Type":"ContainerStarted","Data":"59218e52c6ab0fdf7e5b1808f2f997c35bcf9adddffb7c6b12a64a191e753b93"}
Oct 30 18:51:04.038341 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.038190    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:04.039118 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.039100    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:04.373379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.373347    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"]
Oct 30 18:51:04.377770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.377751    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:04.381896 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.381875    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2aada387-serving-cert\""
Oct 30 18:51:04.390383 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.390361    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"]
Oct 30 18:51:04.403096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.403069    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c57924e-5aff-4907-b39e-4b714ad69474-openshift-service-ca\") pod \"maas-api-e2e-mt-2aada387-577477546b-cl7ms\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:04.403253 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.403216    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c57924e-5aff-4907-b39e-4b714ad69474-maas-api-tls\") pod \"maas-api-e2e-mt-2aada387-577477546b-cl7ms\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:04.403363 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.403348    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wkhl\" (UniqueName: \"kubernetes.io/projected/5c57924e-5aff-4907-b39e-4b714ad69474-kube-api-access-4wkhl\") pod \"maas-api-e2e-mt-2aada387-577477546b-cl7ms\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:04.403435 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.403401    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c57924e-5aff-4907-b39e-4b714ad69474-metrics-certs\") pod \"maas-api-e2e-mt-2aada387-577477546b-cl7ms\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:04.504595 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.504563    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c57924e-5aff-4907-b39e-4b714ad69474-maas-api-tls\") pod \"maas-api-e2e-mt-2aada387-577477546b-cl7ms\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:04.504789 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.504689    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wkhl\" (UniqueName: \"kubernetes.io/projected/5c57924e-5aff-4907-b39e-4b714ad69474-kube-api-access-4wkhl\") pod \"maas-api-e2e-mt-2aada387-577477546b-cl7ms\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:04.504789 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.504738    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c57924e-5aff-4907-b39e-4b714ad69474-metrics-certs\") pod \"maas-api-e2e-mt-2aada387-577477546b-cl7ms\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:04.504912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.504840    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c57924e-5aff-4907-b39e-4b714ad69474-openshift-service-ca\") pod \"maas-api-e2e-mt-2aada387-577477546b-cl7ms\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:04.505530 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.505506    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c57924e-5aff-4907-b39e-4b714ad69474-openshift-service-ca\") pod \"maas-api-e2e-mt-2aada387-577477546b-cl7ms\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:04.507170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.507111    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c57924e-5aff-4907-b39e-4b714ad69474-maas-api-tls\") pod \"maas-api-e2e-mt-2aada387-577477546b-cl7ms\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:04.507288 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.507240    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c57924e-5aff-4907-b39e-4b714ad69474-metrics-certs\") pod \"maas-api-e2e-mt-2aada387-577477546b-cl7ms\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:04.532791 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.532768    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wkhl\" (UniqueName: \"kubernetes.io/projected/5c57924e-5aff-4907-b39e-4b714ad69474-kube-api-access-4wkhl\") pod \"maas-api-e2e-mt-2aada387-577477546b-cl7ms\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:04.696500 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:04.696408    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:05.020344 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:05.020314    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"]
Oct 30 18:51:05.020453 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:51:05.020353    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c57924e_5aff_4907_b39e_4b714ad69474.slice/crio-fdfbfb05a6577ef0ec5600f46e1a5fc1b16ef2c7729dd0ac21e20e1beb207df9 WatchSource:0}: Error finding container fdfbfb05a6577ef0ec5600f46e1a5fc1b16ef2c7729dd0ac21e20e1beb207df9: Status 404 returned error can't find the container with id fdfbfb05a6577ef0ec5600f46e1a5fc1b16ef2c7729dd0ac21e20e1beb207df9
Oct 30 18:51:05.046457 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:05.046428    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h" event={"ID":"5a4347ef-492a-4569-a344-1e683419ea7f","Type":"ContainerStarted","Data":"d5a3a7c509e4450c64837712c5c3c33c35c3f210794aea437c3b46f44e2e9137"}
Oct 30 18:51:05.047903 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:05.047879    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms" event={"ID":"5c57924e-5aff-4907-b39e-4b714ad69474","Type":"ContainerStarted","Data":"fdfbfb05a6577ef0ec5600f46e1a5fc1b16ef2c7729dd0ac21e20e1beb207df9"}
Oct 30 18:51:05.071167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:05.071097    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h" podStartSLOduration=3.071081603 podStartE2EDuration="3.071081603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:05.070171857 +0000 UTC m=+1078.938590190" watchObservedRunningTime="2026-10-30 18:51:05.071081603 +0000 UTC m=+1078.939499934"
Oct 30 18:51:05.353153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:05.342696    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:05.385815 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:05.385775    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:06.056904 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:06.056867    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms" event={"ID":"5c57924e-5aff-4907-b39e-4b714ad69474","Type":"ContainerStarted","Data":"15c810760e657a98732fff443916696f1fa6f0393362cb419ad229a352dd13e2"}
Oct 30 18:51:06.057414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:06.057161    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:06.057414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:06.057204    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:06.058018 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:06.058000    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:51:06.084315 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:06.084274    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms" podStartSLOduration=1.588264692 podStartE2EDuration="2.084261318s" totalImagesPullingTime="495.996626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:06.080731653 +0000 UTC m=+1079.949149983" watchObservedRunningTime="2026-10-30 18:51:06.084261318 +0000 UTC m=+1079.952679648"
Oct 30 18:51:09.073819 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:09.073780    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"]
Oct 30 18:51:09.080514 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:09.080483    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"
Oct 30 18:51:09.090697 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:09.090675    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"]
Oct 30 18:51:09.093590 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:09.093566    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5ad713-dockercfg-qdvjx\""
Oct 30 18:51:09.130296 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:09.130241    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdq6r\" (UniqueName: \"kubernetes.io/projected/bfcd005a-4205-4e87-9e14-c4e4017d3342-kube-api-access-gdq6r\") pod \"payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7\" (UID: \"bfcd005a-4205-4e87-9e14-c4e4017d3342\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"
Oct 30 18:51:09.130692 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:09.130358    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfcd005a-4205-4e87-9e14-c4e4017d3342-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7\" (UID: \"bfcd005a-4205-4e87-9e14-c4e4017d3342\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"
Oct 30 18:51:09.154812 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:09.154789    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5ad713\""
Oct 30 18:51:09.232005 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:09.231963    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdq6r\" (UniqueName: \"kubernetes.io/projected/bfcd005a-4205-4e87-9e14-c4e4017d3342-kube-api-access-gdq6r\") pod \"payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7\" (UID: \"bfcd005a-4205-4e87-9e14-c4e4017d3342\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"
Oct 30 18:51:09.232211 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:09.232050    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfcd005a-4205-4e87-9e14-c4e4017d3342-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7\" (UID: \"bfcd005a-4205-4e87-9e14-c4e4017d3342\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"
Oct 30 18:51:09.232807 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:09.232781    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfcd005a-4205-4e87-9e14-c4e4017d3342-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7\" (UID: \"bfcd005a-4205-4e87-9e14-c4e4017d3342\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"
Oct 30 18:51:09.245509 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:09.245489    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdq6r\" (UniqueName: \"kubernetes.io/projected/bfcd005a-4205-4e87-9e14-c4e4017d3342-kube-api-access-gdq6r\") pod \"payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7\" (UID: \"bfcd005a-4205-4e87-9e14-c4e4017d3342\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"
Oct 30 18:51:09.403154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:09.400367    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"
Oct 30 18:51:09.880903 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:09.880878    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"]
Oct 30 18:51:09.883396 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:51:09.883367    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcd005a_4205_4e87_9e14_c4e4017d3342.slice/crio-1f0f6fa71916f9f59de688ac9546216a8387d06148cdceec32b7ad1a681989df WatchSource:0}: Error finding container 1f0f6fa71916f9f59de688ac9546216a8387d06148cdceec32b7ad1a681989df: Status 404 returned error can't find the container with id 1f0f6fa71916f9f59de688ac9546216a8387d06148cdceec32b7ad1a681989df
Oct 30 18:51:10.087457 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:10.087431    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7" event={"ID":"bfcd005a-4205-4e87-9e14-c4e4017d3342","Type":"ContainerStarted","Data":"a7148f369733a4441cec2a6b18295d272109dc00ecdd5c8f85fd6d1dba98b05a"}
Oct 30 18:51:10.087771 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:10.087478    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7" event={"ID":"bfcd005a-4205-4e87-9e14-c4e4017d3342","Type":"ContainerStarted","Data":"1f0f6fa71916f9f59de688ac9546216a8387d06148cdceec32b7ad1a681989df"}
Oct 30 18:51:10.087771 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:10.087656    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"
Oct 30 18:51:10.108875 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:10.108815    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7" podStartSLOduration=1.1088022180000001 podStartE2EDuration="1.108802218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:10.108055985 +0000 UTC m=+1083.976474315" watchObservedRunningTime="2026-10-30 18:51:10.108802218 +0000 UTC m=+1083.977220563"
Oct 30 18:51:12.014881 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.014847    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"
Oct 30 18:51:12.069988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.069952    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:12.562041 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.562008    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"]
Oct 30 18:51:12.572847 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.572816    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:12.583439 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.583413    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"]
Oct 30 18:51:12.590572 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.590550    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-52fa8b-serving-cert\""
Oct 30 18:51:12.682145 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.682097    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/239a7d10-93ea-4552-bd23-50bcda6292e5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:12.682323 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.682168    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/239a7d10-93ea-4552-bd23-50bcda6292e5-metrics-certs\") pod \"maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:12.682405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.682370    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/239a7d10-93ea-4552-bd23-50bcda6292e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:12.682519 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.682503    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ptrg\" (UniqueName: \"kubernetes.io/projected/239a7d10-93ea-4552-bd23-50bcda6292e5-kube-api-access-9ptrg\") pod \"maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:12.788150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.786656    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/239a7d10-93ea-4552-bd23-50bcda6292e5-metrics-certs\") pod \"maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:12.788150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.786818    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/239a7d10-93ea-4552-bd23-50bcda6292e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:12.788150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.787003    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ptrg\" (UniqueName: \"kubernetes.io/projected/239a7d10-93ea-4552-bd23-50bcda6292e5-kube-api-access-9ptrg\") pod \"maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:12.788150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.787064    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/239a7d10-93ea-4552-bd23-50bcda6292e5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:12.788823 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.788787    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/239a7d10-93ea-4552-bd23-50bcda6292e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:12.825199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.813818    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/239a7d10-93ea-4552-bd23-50bcda6292e5-metrics-certs\") pod \"maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:12.827599 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.827523    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/239a7d10-93ea-4552-bd23-50bcda6292e5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:12.835888 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.835857    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ptrg\" (UniqueName: \"kubernetes.io/projected/239a7d10-93ea-4552-bd23-50bcda6292e5-kube-api-access-9ptrg\") pod \"maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:12.897034 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:12.897003    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:13.030401 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.030370    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"
Oct 30 18:51:13.187231 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.187195    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"]
Oct 30 18:51:13.188048 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:51:13.188022    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239a7d10_93ea_4552_bd23_50bcda6292e5.slice/crio-3a9af40e8017bd951111f1b4d6068f9af3e42178ef80c85c3d8f925a02b8d5c2 WatchSource:0}: Error finding container 3a9af40e8017bd951111f1b4d6068f9af3e42178ef80c85c3d8f925a02b8d5c2: Status 404 returned error can't find the container with id 3a9af40e8017bd951111f1b4d6068f9af3e42178ef80c85c3d8f925a02b8d5c2
Oct 30 18:51:13.518848 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.518760    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"]
Oct 30 18:51:13.525523 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.525495    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:13.536679 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.535039    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"]
Oct 30 18:51:13.536679 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.535911    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0d540b-serving-cert\""
Oct 30 18:51:13.614928 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.614880    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1fbc536-f3af-43db-8bf9-067a4c6598c8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:13.615090 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.614952    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh6tf\" (UniqueName: \"kubernetes.io/projected/d1fbc536-f3af-43db-8bf9-067a4c6598c8-kube-api-access-hh6tf\") pod \"maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:13.615090 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.615036    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1fbc536-f3af-43db-8bf9-067a4c6598c8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:13.615257 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.615235    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1fbc536-f3af-43db-8bf9-067a4c6598c8-metrics-certs\") pod \"maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:13.717163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.716862    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1fbc536-f3af-43db-8bf9-067a4c6598c8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:13.717163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.716937    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6tf\" (UniqueName: \"kubernetes.io/projected/d1fbc536-f3af-43db-8bf9-067a4c6598c8-kube-api-access-hh6tf\") pod \"maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:13.717163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.716967    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1fbc536-f3af-43db-8bf9-067a4c6598c8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:13.717467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.717207    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1fbc536-f3af-43db-8bf9-067a4c6598c8-metrics-certs\") pod \"maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:13.725157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.718183    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1fbc536-f3af-43db-8bf9-067a4c6598c8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:13.738916 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.736006    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1fbc536-f3af-43db-8bf9-067a4c6598c8-metrics-certs\") pod \"maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:13.750671 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.746939    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1fbc536-f3af-43db-8bf9-067a4c6598c8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:13.751667 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.751639    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6tf\" (UniqueName: \"kubernetes.io/projected/d1fbc536-f3af-43db-8bf9-067a4c6598c8-kube-api-access-hh6tf\") pod \"maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:13.848094 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:13.848073    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:14.142161 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:51:14.142088    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1fbc536_f3af_43db_8bf9_067a4c6598c8.slice/crio-642b96dc5b4b58e149caee934b0f11e2189768800d9189623cc920e46e4dab59 WatchSource:0}: Error finding container 642b96dc5b4b58e149caee934b0f11e2189768800d9189623cc920e46e4dab59: Status 404 returned error can't find the container with id 642b96dc5b4b58e149caee934b0f11e2189768800d9189623cc920e46e4dab59
Oct 30 18:51:14.142590 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:14.142259    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"]
Oct 30 18:51:14.155827 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:14.155799    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8" event={"ID":"d1fbc536-f3af-43db-8bf9-067a4c6598c8","Type":"ContainerStarted","Data":"642b96dc5b4b58e149caee934b0f11e2189768800d9189623cc920e46e4dab59"}
Oct 30 18:51:14.157591 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:14.157571    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j" event={"ID":"239a7d10-93ea-4552-bd23-50bcda6292e5","Type":"ContainerStarted","Data":"c25d4676e6d07348c0e35fd7c0536026a970440f7b6831c4405ae80c081ebf46"}
Oct 30 18:51:14.157677 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:14.157618    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j" event={"ID":"239a7d10-93ea-4552-bd23-50bcda6292e5","Type":"ContainerStarted","Data":"3a9af40e8017bd951111f1b4d6068f9af3e42178ef80c85c3d8f925a02b8d5c2"}
Oct 30 18:51:14.157797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:14.157782    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:14.178887 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:14.178841    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j" podStartSLOduration=1.56987635 podStartE2EDuration="2.178825081s" totalImagesPullingTime="608.948731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:14.178430827 +0000 UTC m=+1088.046849172" watchObservedRunningTime="2026-10-30 18:51:14.178825081 +0000 UTC m=+1088.047243414"
Oct 30 18:51:15.165933 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:15.165896    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8" event={"ID":"d1fbc536-f3af-43db-8bf9-067a4c6598c8","Type":"ContainerStarted","Data":"8c028cd05685b29b46e44f7120f5b8df49ae90e006167d2bd5952f4ce65b3d14"}
Oct 30 18:51:15.166355 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:15.165964    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:15.187727 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:15.187676    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8" podStartSLOduration=1.7811641 podStartE2EDuration="2.187661684s" totalImagesPullingTime="406.497584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:15.1868393 +0000 UTC m=+1089.055257649" watchObservedRunningTime="2026-10-30 18:51:15.187661684 +0000 UTC m=+1089.056080015"
Oct 30 18:51:20.171341 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:20.171315    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:51:21.096669 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:21.096644    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"
Oct 30 18:51:21.178031 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:21.178004    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:51:21.656926 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:21.656899    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7411-b6f3dc"]
Oct 30 18:51:21.660820 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:21.660804    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7411-b6f3dc"
Oct 30 18:51:21.667016 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:21.666992    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7h66b\""
Oct 30 18:51:21.667016 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:21.667005    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 18:51:21.667271 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:21.667104    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 18:51:21.668987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:21.668967    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7411-b6f3dc"]
Oct 30 18:51:21.825892 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:21.825860    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hg7m\" (UniqueName: \"kubernetes.io/projected/f853d33a-ed8f-4a33-bbeb-272b7cf68df6-kube-api-access-5hg7m\") pod \"test-curl-7411-b6f3dc\" (UID: \"f853d33a-ed8f-4a33-bbeb-272b7cf68df6\") " pod="opendatahub/test-curl-7411-b6f3dc"
Oct 30 18:51:21.927286 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:21.927222    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hg7m\" (UniqueName: \"kubernetes.io/projected/f853d33a-ed8f-4a33-bbeb-272b7cf68df6-kube-api-access-5hg7m\") pod \"test-curl-7411-b6f3dc\" (UID: \"f853d33a-ed8f-4a33-bbeb-272b7cf68df6\") " pod="opendatahub/test-curl-7411-b6f3dc"
Oct 30 18:51:21.940934 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:21.940905    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hg7m\" (UniqueName: \"kubernetes.io/projected/f853d33a-ed8f-4a33-bbeb-272b7cf68df6-kube-api-access-5hg7m\") pod \"test-curl-7411-b6f3dc\" (UID: \"f853d33a-ed8f-4a33-bbeb-272b7cf68df6\") " pod="opendatahub/test-curl-7411-b6f3dc"
Oct 30 18:51:21.975608 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:21.975591    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7411-b6f3dc"
Oct 30 18:51:22.136205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:22.136175    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:51:22.259477 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:51:22.259448    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf853d33a_ed8f_4a33_bbeb_272b7cf68df6.slice/crio-dc4033f251d1e6caa2fe690ce2e0f413a1c292cf71854dde9fa7e554a199bbc0 WatchSource:0}: Error finding container dc4033f251d1e6caa2fe690ce2e0f413a1c292cf71854dde9fa7e554a199bbc0: Status 404 returned error can't find the container with id dc4033f251d1e6caa2fe690ce2e0f413a1c292cf71854dde9fa7e554a199bbc0
Oct 30 18:51:22.268774 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:22.268752    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7411-b6f3dc"]
Oct 30 18:51:22.302973 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:22.302829    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7411-b6f3dc" event={"ID":"f853d33a-ed8f-4a33-bbeb-272b7cf68df6","Type":"ContainerStarted","Data":"dc4033f251d1e6caa2fe690ce2e0f413a1c292cf71854dde9fa7e554a199bbc0"}
Oct 30 18:51:23.344149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:23.344058    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7411-b6f3dc" event={"ID":"f853d33a-ed8f-4a33-bbeb-272b7cf68df6","Type":"ContainerStarted","Data":"8bf34045b97a906c8eaaaab727152ff9ce21bd2c077705616f32df0d655583e6"}
Oct 30 18:51:23.422150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:23.419485    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7411-b6f3dc" podStartSLOduration=2.419463339 podStartE2EDuration="2.419463339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:23.367286151 +0000 UTC m=+1097.235704482" watchObservedRunningTime="2026-10-30 18:51:23.419463339 +0000 UTC m=+1097.287881670"
Oct 30 18:51:24.153909 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:24.153863    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7411-b6f3dc"]
Oct 30 18:51:24.159649 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:24.159622    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7411-b6f3dc"]
Oct 30 18:51:25.435629 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:25.435590    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7411-b6f3dc" podUID="f853d33a-ed8f-4a33-bbeb-272b7cf68df6" containerName="test-curl-7411-b6f3dc" containerID="cri-o://8bf34045b97a906c8eaaaab727152ff9ce21bd2c077705616f32df0d655583e6" gracePeriod=2
Oct 30 18:51:27.716434 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:27.716228    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7411-b6f3dc"
Oct 30 18:51:27.718517 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:51:27.718478    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:51:27.725084 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:27.725059    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f853d33a-ed8f-4a33-bbeb-272b7cf68df6-kube-api-access-5hg7m\" (UniqueName: \"kubernetes.io/projected/f853d33a-ed8f-4a33-bbeb-272b7cf68df6-kube-api-access-5hg7m\") pod \"f853d33a-ed8f-4a33-bbeb-272b7cf68df6\" (UID: \"f853d33a-ed8f-4a33-bbeb-272b7cf68df6\") "
Oct 30 18:51:27.727265 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:27.727240    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f853d33a-ed8f-4a33-bbeb-272b7cf68df6-kube-api-access-5hg7m" pod "f853d33a-ed8f-4a33-bbeb-272b7cf68df6" (UID: "f853d33a-ed8f-4a33-bbeb-272b7cf68df6"). InnerVolumeSpecName "kube-api-access-5hg7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:27.826653 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:27.826621    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hg7m\" (UniqueName: \"kubernetes.io/projected/f853d33a-ed8f-4a33-bbeb-272b7cf68df6-kube-api-access-5hg7m\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:28.270566 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:28.270528    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f853d33a-ed8f-4a33-bbeb-272b7cf68df6" path="/var/lib/kubelet/pods/f853d33a-ed8f-4a33-bbeb-272b7cf68df6/volumes"
Oct 30 18:51:28.467765 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:28.467732    4372 generic.go:356] "Generic (PLEG): container finished" podID="f853d33a-ed8f-4a33-bbeb-272b7cf68df6" containerID="8bf34045b97a906c8eaaaab727152ff9ce21bd2c077705616f32df0d655583e6" exitCode=137
Oct 30 18:51:28.467966 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:28.467783    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7411-b6f3dc"
Oct 30 18:51:28.467966 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:28.467824    4372 scope.go:122] "RemoveContainer" containerID="8bf34045b97a906c8eaaaab727152ff9ce21bd2c077705616f32df0d655583e6"
Oct 30 18:51:28.477596 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:28.477578    4372 scope.go:122] "RemoveContainer" containerID="8bf34045b97a906c8eaaaab727152ff9ce21bd2c077705616f32df0d655583e6"
Oct 30 18:51:28.477878 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:51:28.477859    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf34045b97a906c8eaaaab727152ff9ce21bd2c077705616f32df0d655583e6\": container with ID starting with 8bf34045b97a906c8eaaaab727152ff9ce21bd2c077705616f32df0d655583e6 not found: ID does not exist" containerID="8bf34045b97a906c8eaaaab727152ff9ce21bd2c077705616f32df0d655583e6"
Oct 30 18:51:28.477923 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:28.477886    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf34045b97a906c8eaaaab727152ff9ce21bd2c077705616f32df0d655583e6"} err="failed to get container status \"8bf34045b97a906c8eaaaab727152ff9ce21bd2c077705616f32df0d655583e6\": rpc error: code = NotFound desc = could not find container \"8bf34045b97a906c8eaaaab727152ff9ce21bd2c077705616f32df0d655583e6\": container with ID starting with 8bf34045b97a906c8eaaaab727152ff9ce21bd2c077705616f32df0d655583e6 not found: ID does not exist"
Oct 30 18:51:32.811804 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:32.811760    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"]
Oct 30 18:51:34.721287 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:34.721235    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"]
Oct 30 18:51:34.779582 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:34.779523    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"]
Oct 30 18:51:34.971045 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:34.964578    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": read tcp 10.133.0.2:47622->10.133.0.100:15021: read: connection reset by peer" start-of-body=
Oct 30 18:51:34.971045 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:34.964644    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t" podUID="0328c404-96a0-447a-b06c-a1ef68ba3178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": read tcp 10.133.0.2:47622->10.133.0.100:15021: read: connection reset by peer"
Oct 30 18:51:35.001052 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.001026    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"]
Oct 30 18:51:35.001325 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.001301    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms" podUID="5c57924e-5aff-4907-b39e-4b714ad69474" containerName="maas-api" containerID="cri-o://15c810760e657a98732fff443916696f1fa6f0393362cb419ad229a352dd13e2" gracePeriod=30
Oct 30 18:51:35.534681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.534565    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:35.592988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.590286    4372 generic.go:356] "Generic (PLEG): container finished" podID="0328c404-96a0-447a-b06c-a1ef68ba3178" containerID="cdf726eec866bac5ec4c9315877bb58e09621cda2eca0f6c018ed444b0910714" exitCode=0
Oct 30 18:51:35.592988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.590489    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t" event={"ID":"0328c404-96a0-447a-b06c-a1ef68ba3178","Type":"ContainerDied","Data":"cdf726eec866bac5ec4c9315877bb58e09621cda2eca0f6c018ed444b0910714"}
Oct 30 18:51:35.592988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.591019    4372 scope.go:122] "RemoveContainer" containerID="cdf726eec866bac5ec4c9315877bb58e09621cda2eca0f6c018ed444b0910714"
Oct 30 18:51:35.596982 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.595412    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:35.596982 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.595506    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:35.596982 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.595554    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:35.601932 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.600274    4372 generic.go:356] "Generic (PLEG): container finished" podID="5c57924e-5aff-4907-b39e-4b714ad69474" containerID="15c810760e657a98732fff443916696f1fa6f0393362cb419ad229a352dd13e2" exitCode=0
Oct 30 18:51:35.601932 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.600423    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms" event={"ID":"5c57924e-5aff-4907-b39e-4b714ad69474","Type":"ContainerDied","Data":"15c810760e657a98732fff443916696f1fa6f0393362cb419ad229a352dd13e2"}
Oct 30 18:51:35.601932 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.600472    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms" event={"ID":"5c57924e-5aff-4907-b39e-4b714ad69474","Type":"ContainerDied","Data":"fdfbfb05a6577ef0ec5600f46e1a5fc1b16ef2c7729dd0ac21e20e1beb207df9"}
Oct 30 18:51:35.601932 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.600502    4372 scope.go:122] "RemoveContainer" containerID="15c810760e657a98732fff443916696f1fa6f0393362cb419ad229a352dd13e2"
Oct 30 18:51:35.601932 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.600669    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"
Oct 30 18:51:35.644737 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.642989    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c57924e-5aff-4907-b39e-4b714ad69474-kube-api-access-4wkhl\" (UniqueName: \"kubernetes.io/projected/5c57924e-5aff-4907-b39e-4b714ad69474-kube-api-access-4wkhl\") pod \"5c57924e-5aff-4907-b39e-4b714ad69474\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") "
Oct 30 18:51:35.644737 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.643107    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c57924e-5aff-4907-b39e-4b714ad69474-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c57924e-5aff-4907-b39e-4b714ad69474-maas-api-tls\") pod \"5c57924e-5aff-4907-b39e-4b714ad69474\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") "
Oct 30 18:51:35.644737 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.643206    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c57924e-5aff-4907-b39e-4b714ad69474-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c57924e-5aff-4907-b39e-4b714ad69474-metrics-certs\") pod \"5c57924e-5aff-4907-b39e-4b714ad69474\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") "
Oct 30 18:51:35.644737 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.643322    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c57924e-5aff-4907-b39e-4b714ad69474-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c57924e-5aff-4907-b39e-4b714ad69474-openshift-service-ca\") pod \"5c57924e-5aff-4907-b39e-4b714ad69474\" (UID: \"5c57924e-5aff-4907-b39e-4b714ad69474\") "
Oct 30 18:51:35.648464 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.647009    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c57924e-5aff-4907-b39e-4b714ad69474-openshift-service-ca" pod "5c57924e-5aff-4907-b39e-4b714ad69474" (UID: "5c57924e-5aff-4907-b39e-4b714ad69474"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:35.651724 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.651667    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c57924e-5aff-4907-b39e-4b714ad69474-kube-api-access-4wkhl" pod "5c57924e-5aff-4907-b39e-4b714ad69474" (UID: "5c57924e-5aff-4907-b39e-4b714ad69474"). InnerVolumeSpecName "kube-api-access-4wkhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:35.652434 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.652390    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c57924e-5aff-4907-b39e-4b714ad69474-maas-api-tls" pod "5c57924e-5aff-4907-b39e-4b714ad69474" (UID: "5c57924e-5aff-4907-b39e-4b714ad69474"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:35.658067 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.658010    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c57924e-5aff-4907-b39e-4b714ad69474-metrics-certs" pod "5c57924e-5aff-4907-b39e-4b714ad69474" (UID: "5c57924e-5aff-4907-b39e-4b714ad69474"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:35.660709 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.660501    4372 scope.go:122] "RemoveContainer" containerID="15c810760e657a98732fff443916696f1fa6f0393362cb419ad229a352dd13e2"
Oct 30 18:51:35.660969 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:51:35.660933    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c810760e657a98732fff443916696f1fa6f0393362cb419ad229a352dd13e2\": container with ID starting with 15c810760e657a98732fff443916696f1fa6f0393362cb419ad229a352dd13e2 not found: ID does not exist" containerID="15c810760e657a98732fff443916696f1fa6f0393362cb419ad229a352dd13e2"
Oct 30 18:51:35.661070 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.660973    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c810760e657a98732fff443916696f1fa6f0393362cb419ad229a352dd13e2"} err="failed to get container status \"15c810760e657a98732fff443916696f1fa6f0393362cb419ad229a352dd13e2\": rpc error: code = NotFound desc = could not find container \"15c810760e657a98732fff443916696f1fa6f0393362cb419ad229a352dd13e2\": container with ID starting with 15c810760e657a98732fff443916696f1fa6f0393362cb419ad229a352dd13e2 not found: ID does not exist"
Oct 30 18:51:35.751704 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.751632    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c57924e-5aff-4907-b39e-4b714ad69474-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:35.751704 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.751669    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c57924e-5aff-4907-b39e-4b714ad69474-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:35.752297 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.751720    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wkhl\" (UniqueName: \"kubernetes.io/projected/5c57924e-5aff-4907-b39e-4b714ad69474-kube-api-access-4wkhl\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:35.752297 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.751737    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c57924e-5aff-4907-b39e-4b714ad69474-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:35.928252 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.928218    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"]
Oct 30 18:51:35.933463 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:35.933438    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-cl7ms"]
Oct 30 18:51:36.271961 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:36.271879    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c57924e-5aff-4907-b39e-4b714ad69474" path="/var/lib/kubelet/pods/5c57924e-5aff-4907-b39e-4b714ad69474/volumes"
Oct 30 18:51:36.611396 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:36.611358    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t" event={"ID":"0328c404-96a0-447a-b06c-a1ef68ba3178","Type":"ContainerStarted","Data":"f3cc23758dabf5a8524a34d75ff60165d1154fa1296896275ce58f83d820765b"}
Oct 30 18:51:37.558039 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:37.557991    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:37.558840 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:37.558816    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:51:37.559039 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:37.559013    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t" podUID="0328c404-96a0-447a-b06c-a1ef68ba3178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 18:51:37.813479 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:37.813381    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7" podUID="bfcd005a-4205-4e87-9e14-c4e4017d3342" containerName="payload-pre-processing" containerID="cri-o://a7148f369733a4441cec2a6b18295d272109dc00ecdd5c8f85fd6d1dba98b05a" gracePeriod=25
Oct 30 18:51:38.149759 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.149736    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"
Oct 30 18:51:38.328934 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.328894    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfcd005a-4205-4e87-9e14-c4e4017d3342-kube-api-access-gdq6r\" (UniqueName: \"kubernetes.io/projected/bfcd005a-4205-4e87-9e14-c4e4017d3342-kube-api-access-gdq6r\") pod \"bfcd005a-4205-4e87-9e14-c4e4017d3342\" (UID: \"bfcd005a-4205-4e87-9e14-c4e4017d3342\") "
Oct 30 18:51:38.329175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.329029    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfcd005a-4205-4e87-9e14-c4e4017d3342-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfcd005a-4205-4e87-9e14-c4e4017d3342-plugins-config-volume\") pod \"bfcd005a-4205-4e87-9e14-c4e4017d3342\" (UID: \"bfcd005a-4205-4e87-9e14-c4e4017d3342\") "
Oct 30 18:51:38.329432 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.329408    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfcd005a-4205-4e87-9e14-c4e4017d3342-plugins-config-volume" pod "bfcd005a-4205-4e87-9e14-c4e4017d3342" (UID: "bfcd005a-4205-4e87-9e14-c4e4017d3342"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:38.329719 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.329696    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfcd005a-4205-4e87-9e14-c4e4017d3342-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:38.333380 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.333355    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfcd005a-4205-4e87-9e14-c4e4017d3342-kube-api-access-gdq6r" pod "bfcd005a-4205-4e87-9e14-c4e4017d3342" (UID: "bfcd005a-4205-4e87-9e14-c4e4017d3342"). InnerVolumeSpecName "kube-api-access-gdq6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:38.431490 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.431390    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdq6r\" (UniqueName: \"kubernetes.io/projected/bfcd005a-4205-4e87-9e14-c4e4017d3342-kube-api-access-gdq6r\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:38.553549 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.553519    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:51:38.553724 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.553568    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t" podUID="0328c404-96a0-447a-b06c-a1ef68ba3178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 18:51:38.669977 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.669938    4372 generic.go:356] "Generic (PLEG): container finished" podID="bfcd005a-4205-4e87-9e14-c4e4017d3342" containerID="a7148f369733a4441cec2a6b18295d272109dc00ecdd5c8f85fd6d1dba98b05a" exitCode=0
Oct 30 18:51:38.670564 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.670530    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"
Oct 30 18:51:38.670707 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.670442    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7" event={"ID":"bfcd005a-4205-4e87-9e14-c4e4017d3342","Type":"ContainerDied","Data":"a7148f369733a4441cec2a6b18295d272109dc00ecdd5c8f85fd6d1dba98b05a"}
Oct 30 18:51:38.670872 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.670752    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7" event={"ID":"bfcd005a-4205-4e87-9e14-c4e4017d3342","Type":"ContainerDied","Data":"1f0f6fa71916f9f59de688ac9546216a8387d06148cdceec32b7ad1a681989df"}
Oct 30 18:51:38.670872 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.670785    4372 scope.go:122] "RemoveContainer" containerID="a7148f369733a4441cec2a6b18295d272109dc00ecdd5c8f85fd6d1dba98b05a"
Oct 30 18:51:38.700362 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.691449    4372 scope.go:122] "RemoveContainer" containerID="a7148f369733a4441cec2a6b18295d272109dc00ecdd5c8f85fd6d1dba98b05a"
Oct 30 18:51:38.700362 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:51:38.691769    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7148f369733a4441cec2a6b18295d272109dc00ecdd5c8f85fd6d1dba98b05a\": container with ID starting with a7148f369733a4441cec2a6b18295d272109dc00ecdd5c8f85fd6d1dba98b05a not found: ID does not exist" containerID="a7148f369733a4441cec2a6b18295d272109dc00ecdd5c8f85fd6d1dba98b05a"
Oct 30 18:51:38.700362 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.691805    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7148f369733a4441cec2a6b18295d272109dc00ecdd5c8f85fd6d1dba98b05a"} err="failed to get container status \"a7148f369733a4441cec2a6b18295d272109dc00ecdd5c8f85fd6d1dba98b05a\": rpc error: code = NotFound desc = could not find container \"a7148f369733a4441cec2a6b18295d272109dc00ecdd5c8f85fd6d1dba98b05a\": container with ID starting with a7148f369733a4441cec2a6b18295d272109dc00ecdd5c8f85fd6d1dba98b05a not found: ID does not exist"
Oct 30 18:51:38.716604 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.716573    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"]
Oct 30 18:51:38.720586 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:38.720564    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5ad713-86c6769fc5-h59r7"]
Oct 30 18:51:39.553962 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:39.553920    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:51:39.554112 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:39.553979    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t" podUID="0328c404-96a0-447a-b06c-a1ef68ba3178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 18:51:39.722830 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:39.722788    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w" podUID="494f7ef0-35bc-4c22-85d0-8799b860298e" containerName="payload-processing" containerID="cri-o://24e9c6d6e7d326434c0b5e368c5ed14f6c7c1440140ec839f3b8a48e7813a4cc" gracePeriod=25
Oct 30 18:51:39.782239 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:39.782200    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn" podUID="73af2591-d3cd-42ff-a60a-dfa5107562ed" containerName="payload-pre-processing" containerID="cri-o://ebf8c6f2ae52679a0980b5b12ec2c6f7e9bf34e1b5880720368de2bd58154260" gracePeriod=25
Oct 30 18:51:40.288453 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.286773    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfcd005a-4205-4e87-9e14-c4e4017d3342" path="/var/lib/kubelet/pods/bfcd005a-4205-4e87-9e14-c4e4017d3342/volumes"
Oct 30 18:51:40.325220 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.293119    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"
Oct 30 18:51:40.414901 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.412368    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494f7ef0-35bc-4c22-85d0-8799b860298e-kube-api-access-l2ltj\" (UniqueName: \"kubernetes.io/projected/494f7ef0-35bc-4c22-85d0-8799b860298e-kube-api-access-l2ltj\") pod \"494f7ef0-35bc-4c22-85d0-8799b860298e\" (UID: \"494f7ef0-35bc-4c22-85d0-8799b860298e\") "
Oct 30 18:51:40.414901 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.412639    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494f7ef0-35bc-4c22-85d0-8799b860298e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494f7ef0-35bc-4c22-85d0-8799b860298e-plugins-config-volume\") pod \"494f7ef0-35bc-4c22-85d0-8799b860298e\" (UID: \"494f7ef0-35bc-4c22-85d0-8799b860298e\") "
Oct 30 18:51:40.414901 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.413157    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494f7ef0-35bc-4c22-85d0-8799b860298e-plugins-config-volume" pod "494f7ef0-35bc-4c22-85d0-8799b860298e" (UID: "494f7ef0-35bc-4c22-85d0-8799b860298e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:40.414901 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.414323    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494f7ef0-35bc-4c22-85d0-8799b860298e-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:40.417303 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.417276    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494f7ef0-35bc-4c22-85d0-8799b860298e-kube-api-access-l2ltj" pod "494f7ef0-35bc-4c22-85d0-8799b860298e" (UID: "494f7ef0-35bc-4c22-85d0-8799b860298e"). InnerVolumeSpecName "kube-api-access-l2ltj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:40.443737 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.443717    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"
Oct 30 18:51:40.515610 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.515580    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73af2591-d3cd-42ff-a60a-dfa5107562ed-kube-api-access-hs6w5\" (UniqueName: \"kubernetes.io/projected/73af2591-d3cd-42ff-a60a-dfa5107562ed-kube-api-access-hs6w5\") pod \"73af2591-d3cd-42ff-a60a-dfa5107562ed\" (UID: \"73af2591-d3cd-42ff-a60a-dfa5107562ed\") "
Oct 30 18:51:40.515795 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.515783    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73af2591-d3cd-42ff-a60a-dfa5107562ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73af2591-d3cd-42ff-a60a-dfa5107562ed-plugins-config-volume\") pod \"73af2591-d3cd-42ff-a60a-dfa5107562ed\" (UID: \"73af2591-d3cd-42ff-a60a-dfa5107562ed\") "
Oct 30 18:51:40.516251 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.516233    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2ltj\" (UniqueName: \"kubernetes.io/projected/494f7ef0-35bc-4c22-85d0-8799b860298e-kube-api-access-l2ltj\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:40.516580 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.516507    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73af2591-d3cd-42ff-a60a-dfa5107562ed-plugins-config-volume" pod "73af2591-d3cd-42ff-a60a-dfa5107562ed" (UID: "73af2591-d3cd-42ff-a60a-dfa5107562ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:40.518618 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.518594    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73af2591-d3cd-42ff-a60a-dfa5107562ed-kube-api-access-hs6w5" pod "73af2591-d3cd-42ff-a60a-dfa5107562ed" (UID: "73af2591-d3cd-42ff-a60a-dfa5107562ed"). InnerVolumeSpecName "kube-api-access-hs6w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:40.562657 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.562627    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:51:40.562769 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.562673    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t" podUID="0328c404-96a0-447a-b06c-a1ef68ba3178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 18:51:40.617403 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.617332    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73af2591-d3cd-42ff-a60a-dfa5107562ed-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:40.617403 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.617357    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs6w5\" (UniqueName: \"kubernetes.io/projected/73af2591-d3cd-42ff-a60a-dfa5107562ed-kube-api-access-hs6w5\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:40.708767 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.708732    4372 generic.go:356] "Generic (PLEG): container finished" podID="494f7ef0-35bc-4c22-85d0-8799b860298e" containerID="24e9c6d6e7d326434c0b5e368c5ed14f6c7c1440140ec839f3b8a48e7813a4cc" exitCode=0
Oct 30 18:51:40.708942 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.708790    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"
Oct 30 18:51:40.708942 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.708806    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w" event={"ID":"494f7ef0-35bc-4c22-85d0-8799b860298e","Type":"ContainerDied","Data":"24e9c6d6e7d326434c0b5e368c5ed14f6c7c1440140ec839f3b8a48e7813a4cc"}
Oct 30 18:51:40.708942 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.708880    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w" event={"ID":"494f7ef0-35bc-4c22-85d0-8799b860298e","Type":"ContainerDied","Data":"600a78147031ad515628662fcd910f46e1ff7f6fcc3028fb0acffbf99005e96b"}
Oct 30 18:51:40.708942 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.708904    4372 scope.go:122] "RemoveContainer" containerID="24e9c6d6e7d326434c0b5e368c5ed14f6c7c1440140ec839f3b8a48e7813a4cc"
Oct 30 18:51:40.710429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.710407    4372 generic.go:356] "Generic (PLEG): container finished" podID="73af2591-d3cd-42ff-a60a-dfa5107562ed" containerID="ebf8c6f2ae52679a0980b5b12ec2c6f7e9bf34e1b5880720368de2bd58154260" exitCode=0
Oct 30 18:51:40.710549 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.710476    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"
Oct 30 18:51:40.710549 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.710488    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn" event={"ID":"73af2591-d3cd-42ff-a60a-dfa5107562ed","Type":"ContainerDied","Data":"ebf8c6f2ae52679a0980b5b12ec2c6f7e9bf34e1b5880720368de2bd58154260"}
Oct 30 18:51:40.710549 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.710531    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn" event={"ID":"73af2591-d3cd-42ff-a60a-dfa5107562ed","Type":"ContainerDied","Data":"ada7ceac1d770ab884ecf4f92a661571fc15e959432c430cb170961275520081"}
Oct 30 18:51:40.719688 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.719654    4372 scope.go:122] "RemoveContainer" containerID="24e9c6d6e7d326434c0b5e368c5ed14f6c7c1440140ec839f3b8a48e7813a4cc"
Oct 30 18:51:40.719980 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:51:40.719954    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e9c6d6e7d326434c0b5e368c5ed14f6c7c1440140ec839f3b8a48e7813a4cc\": container with ID starting with 24e9c6d6e7d326434c0b5e368c5ed14f6c7c1440140ec839f3b8a48e7813a4cc not found: ID does not exist" containerID="24e9c6d6e7d326434c0b5e368c5ed14f6c7c1440140ec839f3b8a48e7813a4cc"
Oct 30 18:51:40.720073 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.719992    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e9c6d6e7d326434c0b5e368c5ed14f6c7c1440140ec839f3b8a48e7813a4cc"} err="failed to get container status \"24e9c6d6e7d326434c0b5e368c5ed14f6c7c1440140ec839f3b8a48e7813a4cc\": rpc error: code = NotFound desc = could not find container \"24e9c6d6e7d326434c0b5e368c5ed14f6c7c1440140ec839f3b8a48e7813a4cc\": container with ID starting with 24e9c6d6e7d326434c0b5e368c5ed14f6c7c1440140ec839f3b8a48e7813a4cc not found: ID does not exist"
Oct 30 18:51:40.720073 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.720016    4372 scope.go:122] "RemoveContainer" containerID="ebf8c6f2ae52679a0980b5b12ec2c6f7e9bf34e1b5880720368de2bd58154260"
Oct 30 18:51:40.730761 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.729949    4372 scope.go:122] "RemoveContainer" containerID="ebf8c6f2ae52679a0980b5b12ec2c6f7e9bf34e1b5880720368de2bd58154260"
Oct 30 18:51:40.730761 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:51:40.730294    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf8c6f2ae52679a0980b5b12ec2c6f7e9bf34e1b5880720368de2bd58154260\": container with ID starting with ebf8c6f2ae52679a0980b5b12ec2c6f7e9bf34e1b5880720368de2bd58154260 not found: ID does not exist" containerID="ebf8c6f2ae52679a0980b5b12ec2c6f7e9bf34e1b5880720368de2bd58154260"
Oct 30 18:51:40.730761 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.730319    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf8c6f2ae52679a0980b5b12ec2c6f7e9bf34e1b5880720368de2bd58154260"} err="failed to get container status \"ebf8c6f2ae52679a0980b5b12ec2c6f7e9bf34e1b5880720368de2bd58154260\": rpc error: code = NotFound desc = could not find container \"ebf8c6f2ae52679a0980b5b12ec2c6f7e9bf34e1b5880720368de2bd58154260\": container with ID starting with ebf8c6f2ae52679a0980b5b12ec2c6f7e9bf34e1b5880720368de2bd58154260 not found: ID does not exist"
Oct 30 18:51:40.744695 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.744676    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"]
Oct 30 18:51:40.751138 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.751107    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-8qwcn"]
Oct 30 18:51:40.762329 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.762309    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"]
Oct 30 18:51:40.767256 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:40.767237    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-89r8w"]
Oct 30 18:51:41.553775 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:41.553744    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:51:41.553958 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:41.553796    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t" podUID="0328c404-96a0-447a-b06c-a1ef68ba3178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 18:51:42.266035 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:42.266004    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494f7ef0-35bc-4c22-85d0-8799b860298e" path="/var/lib/kubelet/pods/494f7ef0-35bc-4c22-85d0-8799b860298e/volumes"
Oct 30 18:51:42.266604 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:42.266588    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73af2591-d3cd-42ff-a60a-dfa5107562ed" path="/var/lib/kubelet/pods/73af2591-d3cd-42ff-a60a-dfa5107562ed/volumes"
Oct 30 18:51:42.557427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:42.557345    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:42.557820 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:42.557807    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:42.558599 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:42.558575    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:51:48.598399 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:48.598323    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:51:53.120221 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120177    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"]
Oct 30 18:51:53.120789 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120774    4372 cpu_manager.go:414] "containerMap: removing container" podUID="bfcd005a-4205-4e87-9e14-c4e4017d3342" containerName="payload-pre-processing"
Oct 30 18:51:53.120834 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120791    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfcd005a-4205-4e87-9e14-c4e4017d3342" containerName="payload-pre-processing"
Oct 30 18:51:53.120834 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120814    4372 cpu_manager.go:414] "containerMap: removing container" podUID="494f7ef0-35bc-4c22-85d0-8799b860298e" containerName="payload-processing"
Oct 30 18:51:53.120834 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120819    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494f7ef0-35bc-4c22-85d0-8799b860298e" containerName="payload-processing"
Oct 30 18:51:53.120834 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120826    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f853d33a-ed8f-4a33-bbeb-272b7cf68df6" containerName="test-curl-7411-b6f3dc"
Oct 30 18:51:53.120834 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120831    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f853d33a-ed8f-4a33-bbeb-272b7cf68df6" containerName="test-curl-7411-b6f3dc"
Oct 30 18:51:53.120987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120841    4372 cpu_manager.go:414] "containerMap: removing container" podUID="5c57924e-5aff-4907-b39e-4b714ad69474" containerName="maas-api"
Oct 30 18:51:53.120987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120846    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c57924e-5aff-4907-b39e-4b714ad69474" containerName="maas-api"
Oct 30 18:51:53.120987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120864    4372 cpu_manager.go:414] "containerMap: removing container" podUID="73af2591-d3cd-42ff-a60a-dfa5107562ed" containerName="payload-pre-processing"
Oct 30 18:51:53.120987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120869    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73af2591-d3cd-42ff-a60a-dfa5107562ed" containerName="payload-pre-processing"
Oct 30 18:51:53.120987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120958    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="73af2591-d3cd-42ff-a60a-dfa5107562ed" containerName="payload-pre-processing"
Oct 30 18:51:53.120987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120967    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="494f7ef0-35bc-4c22-85d0-8799b860298e" containerName="payload-processing"
Oct 30 18:51:53.120987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120974    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c57924e-5aff-4907-b39e-4b714ad69474" containerName="maas-api"
Oct 30 18:51:53.120987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120979    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="f853d33a-ed8f-4a33-bbeb-272b7cf68df6" containerName="test-curl-7411-b6f3dc"
Oct 30 18:51:53.120987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.120985    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfcd005a-4205-4e87-9e14-c4e4017d3342" containerName="payload-pre-processing"
Oct 30 18:51:53.124388 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.124368    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.129678 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.129651    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8fd4ed-openshift-default-dockercfg-klxct\""
Oct 30 18:51:53.143259 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.143236    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"]
Oct 30 18:51:53.229205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.229177    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-token\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.229359 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.229218    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-workload-socket\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.229359 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.229244    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-credential-socket\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.229436 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.229394    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2tps\" (UniqueName: \"kubernetes.io/projected/2313cd49-c455-47e7-94df-091ce8c2edf9-kube-api-access-l2tps\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.229539 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.229524    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2313cd49-c455-47e7-94df-091ce8c2edf9-istiod-ca-cert\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.229572 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.229548    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-podinfo\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.229572 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.229567    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-envoy\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.229637 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.229594    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-workload-certs\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.229720 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.229707    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-data\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.332248 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.332206    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2tps\" (UniqueName: \"kubernetes.io/projected/2313cd49-c455-47e7-94df-091ce8c2edf9-kube-api-access-l2tps\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.333288 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.333259    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2313cd49-c455-47e7-94df-091ce8c2edf9-istiod-ca-cert\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.343823 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.343796    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-podinfo\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.344022 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.344005    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-envoy\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.351156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.344182    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-workload-certs\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.351445 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.351408    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-data\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.351691 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.351672    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-token\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.351836 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.351822    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-workload-socket\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.351957 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.351938    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-credential-socket\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.360149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.347214    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-workload-certs\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.360812 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.360788    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-data\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.360967 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.334873    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2313cd49-c455-47e7-94df-091ce8c2edf9-istiod-ca-cert\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.361621 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.361599    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-workload-socket\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.361988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.361964    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-credential-socket\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.373185 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.367053    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2tps\" (UniqueName: \"kubernetes.io/projected/2313cd49-c455-47e7-94df-091ce8c2edf9-kube-api-access-l2tps\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.389505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.389472    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-envoy\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.397153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.390838    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-token\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.397153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.391004    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-podinfo\") pod \"e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") " pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.441432 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.441397    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:53.847926 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.847896    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"]
Oct 30 18:51:53.849018 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:51:53.848983    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2313cd49_c455_47e7_94df_091ce8c2edf9.slice/crio-c329fe6d7baed360eff04ff3d280dc4317219961c11843abf9d88b63a0d533a5 WatchSource:0}: Error finding container c329fe6d7baed360eff04ff3d280dc4317219961c11843abf9d88b63a0d533a5: Status 404 returned error can't find the container with id c329fe6d7baed360eff04ff3d280dc4317219961c11843abf9d88b63a0d533a5
Oct 30 18:51:53.852499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.852081    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:53.852499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.852174    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:53.852499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.852208    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:53.879483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:53.879456    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" event={"ID":"2313cd49-c455-47e7-94df-091ce8c2edf9","Type":"ContainerStarted","Data":"c329fe6d7baed360eff04ff3d280dc4317219961c11843abf9d88b63a0d533a5"}
Oct 30 18:51:54.921150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:54.919526    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" event={"ID":"2313cd49-c455-47e7-94df-091ce8c2edf9","Type":"ContainerStarted","Data":"2adf405b22604db675e0895e8d6dd795a17d4de6976b9902cef1ed0ec9bf3d2e"}
Oct 30 18:51:54.973045 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:54.970276    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" podStartSLOduration=1.9702612689999999 podStartE2EDuration="1.970261269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:54.966673268 +0000 UTC m=+1128.835091599" watchObservedRunningTime="2026-10-30 18:51:54.970261269 +0000 UTC m=+1128.838679598"
Oct 30 18:51:55.312229 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.311731    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"]
Oct 30 18:51:55.317513 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.317474    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.322810 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.322223    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ecef08-openshift-default-dockercfg-sf886\""
Oct 30 18:51:55.336687 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.336662    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"]
Oct 30 18:51:55.502150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.495972    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-workload-socket\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.502150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.496023    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hln4t\" (UniqueName: \"kubernetes.io/projected/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-kube-api-access-hln4t\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.502150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.496212    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-envoy\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.502150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.496270    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-data\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.502150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.496322    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-credential-socket\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.502150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.496356    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-workload-certs\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.502150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.496398    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istiod-ca-cert\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.502150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.496450    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-podinfo\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.502150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.496496    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-token\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.533150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.518426    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:55.602943 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.602610    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:55.622808 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.622770    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-credential-socket\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.637154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.623573    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-workload-certs\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.637154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.634869    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istiod-ca-cert\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.637154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.634947    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-podinfo\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.637154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.634990    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-token\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.637154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.635116    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-workload-socket\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.637154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.634544    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-workload-certs\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.637154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.635164    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hln4t\" (UniqueName: \"kubernetes.io/projected/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-kube-api-access-hln4t\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.637154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.635450    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-envoy\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.637154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.635528    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-data\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.637154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.635944    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-data\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.637154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.623501    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-credential-socket\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.637154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.636645    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istiod-ca-cert\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.638868 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.638822    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-workload-socket\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.710173 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.663939    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-envoy\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.710173 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.681733    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-token\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.710173 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.689087    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hln4t\" (UniqueName: \"kubernetes.io/projected/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-kube-api-access-hln4t\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.726003 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.725968    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-podinfo\") pod \"e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") " pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.945481 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.945398    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:55.972432 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.972379    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:55.974060 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:55.974035    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:51:56.308692 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:51:56.308654    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbe58a7_df0d_4e2b_a4bf_494ebc67ba04.slice/crio-0d7f2ea94fda9e61c7a3fa459f637fe8db41968ff6757ae49e3550ea07181e38 WatchSource:0}: Error finding container 0d7f2ea94fda9e61c7a3fa459f637fe8db41968ff6757ae49e3550ea07181e38: Status 404 returned error can't find the container with id 0d7f2ea94fda9e61c7a3fa459f637fe8db41968ff6757ae49e3550ea07181e38
Oct 30 18:51:56.310493 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:56.310466    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"]
Oct 30 18:51:56.311537 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:56.311501    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:56.311615 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:56.311566    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:56.311615 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:56.311596    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:56.993150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:56.983426    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc" event={"ID":"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04","Type":"ContainerStarted","Data":"0fb14a8ccf447af93953cf2b5ac08e30bb33616ce4b9d0e69f21b9e723567e34"}
Oct 30 18:51:56.993150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:56.983486    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc" event={"ID":"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04","Type":"ContainerStarted","Data":"0d7f2ea94fda9e61c7a3fa459f637fe8db41968ff6757ae49e3550ea07181e38"}
Oct 30 18:51:57.015254 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:57.015204    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc" podStartSLOduration=2.015189712 podStartE2EDuration="2.015189712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:57.011326899 +0000 UTC m=+1130.879745254" watchObservedRunningTime="2026-10-30 18:51:57.015189712 +0000 UTC m=+1130.883608041"
Oct 30 18:51:57.809384 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:51:57.809341    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:51:57.951023 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:57.950984    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:57.956783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:57.956755    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:58.013881 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:58.013851    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:58.014781 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:58.014760    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:51:58.722893 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:58.722860    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"]
Oct 30 18:51:58.727354 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:58.727331    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"
Oct 30 18:51:58.741537 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:58.741513    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8fd4ed\""
Oct 30 18:51:58.742602 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:58.742581    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"]
Oct 30 18:51:58.829724 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:58.829691    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpjtf\" (UniqueName: \"kubernetes.io/projected/58266961-626d-4032-a62d-1a080000767f-kube-api-access-zpjtf\") pod \"payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx\" (UID: \"58266961-626d-4032-a62d-1a080000767f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"
Oct 30 18:51:58.829977 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:58.829956    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58266961-626d-4032-a62d-1a080000767f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx\" (UID: \"58266961-626d-4032-a62d-1a080000767f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"
Oct 30 18:51:58.978369 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:58.978188    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"]
Oct 30 18:51:59.054211 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:58.978478    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58266961-626d-4032-a62d-1a080000767f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx\" (UID: \"58266961-626d-4032-a62d-1a080000767f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"
Oct 30 18:51:59.067396 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:59.067356    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpjtf\" (UniqueName: \"kubernetes.io/projected/58266961-626d-4032-a62d-1a080000767f-kube-api-access-zpjtf\") pod \"payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx\" (UID: \"58266961-626d-4032-a62d-1a080000767f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"
Oct 30 18:51:59.070610 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:58.979362    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58266961-626d-4032-a62d-1a080000767f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx\" (UID: \"58266961-626d-4032-a62d-1a080000767f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"
Oct 30 18:51:59.214152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:59.213316    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"]
Oct 30 18:51:59.214152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:59.213451    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"
Oct 30 18:51:59.234198 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:59.230478    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8fd4ed-dockercfg-f6c2m\""
Oct 30 18:51:59.277467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:59.277428    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpjtf\" (UniqueName: \"kubernetes.io/projected/58266961-626d-4032-a62d-1a080000767f-kube-api-access-zpjtf\") pod \"payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx\" (UID: \"58266961-626d-4032-a62d-1a080000767f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"
Oct 30 18:51:59.300393 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:59.297612    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2809d5-9d1e-4d21-baa5-9c38079d5afc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7\" (UID: \"3d2809d5-9d1e-4d21-baa5-9c38079d5afc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"
Oct 30 18:51:59.300393 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:59.297840    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pdkp\" (UniqueName: \"kubernetes.io/projected/3d2809d5-9d1e-4d21-baa5-9c38079d5afc-kube-api-access-9pdkp\") pod \"payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7\" (UID: \"3d2809d5-9d1e-4d21-baa5-9c38079d5afc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"
Oct 30 18:51:59.344826 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:59.344796    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"
Oct 30 18:51:59.406150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:59.402342    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdkp\" (UniqueName: \"kubernetes.io/projected/3d2809d5-9d1e-4d21-baa5-9c38079d5afc-kube-api-access-9pdkp\") pod \"payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7\" (UID: \"3d2809d5-9d1e-4d21-baa5-9c38079d5afc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"
Oct 30 18:51:59.406150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:59.402563    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2809d5-9d1e-4d21-baa5-9c38079d5afc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7\" (UID: \"3d2809d5-9d1e-4d21-baa5-9c38079d5afc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"
Oct 30 18:51:59.406150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:59.405492    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2809d5-9d1e-4d21-baa5-9c38079d5afc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7\" (UID: \"3d2809d5-9d1e-4d21-baa5-9c38079d5afc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"
Oct 30 18:51:59.432872 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:59.432820    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pdkp\" (UniqueName: \"kubernetes.io/projected/3d2809d5-9d1e-4d21-baa5-9c38079d5afc-kube-api-access-9pdkp\") pod \"payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7\" (UID: \"3d2809d5-9d1e-4d21-baa5-9c38079d5afc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"
Oct 30 18:51:59.605590 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:59.605552    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"
Oct 30 18:51:59.665722 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:51:59.665679    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58266961_626d_4032_a62d_1a080000767f.slice/crio-1021a7409aec75ec24eb39988ffc78e5de144bf66484b72a31877109b1874675 WatchSource:0}: Error finding container 1021a7409aec75ec24eb39988ffc78e5de144bf66484b72a31877109b1874675: Status 404 returned error can't find the container with id 1021a7409aec75ec24eb39988ffc78e5de144bf66484b72a31877109b1874675
Oct 30 18:51:59.676533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:51:59.676511    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"]
Oct 30 18:52:00.057349 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:00.057321    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"]
Oct 30 18:52:00.058244 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:52:00.058219    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2809d5_9d1e_4d21_baa5_9c38079d5afc.slice/crio-e2d7ea0897a48aa6a25d4a7330d8e1956eaef377ac4c21cfd6a6c5f7c8d0df3e WatchSource:0}: Error finding container e2d7ea0897a48aa6a25d4a7330d8e1956eaef377ac4c21cfd6a6c5f7c8d0df3e: Status 404 returned error can't find the container with id e2d7ea0897a48aa6a25d4a7330d8e1956eaef377ac4c21cfd6a6c5f7c8d0df3e
Oct 30 18:52:00.111464 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:00.111428    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx" event={"ID":"58266961-626d-4032-a62d-1a080000767f","Type":"ContainerStarted","Data":"9fbfcf3a946b61f20ffb0b7290b741a6175177b78d3a77494384b396c2e06600"}
Oct 30 18:52:00.111599 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:00.111487    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx" event={"ID":"58266961-626d-4032-a62d-1a080000767f","Type":"ContainerStarted","Data":"1021a7409aec75ec24eb39988ffc78e5de144bf66484b72a31877109b1874675"}
Oct 30 18:52:00.111599 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:00.111520    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"
Oct 30 18:52:00.112643 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:00.112609    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7" event={"ID":"3d2809d5-9d1e-4d21-baa5-9c38079d5afc","Type":"ContainerStarted","Data":"e2d7ea0897a48aa6a25d4a7330d8e1956eaef377ac4c21cfd6a6c5f7c8d0df3e"}
Oct 30 18:52:00.155955 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:00.155913    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx" podStartSLOduration=2.155900338 podStartE2EDuration="2.155900338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:00.15453791 +0000 UTC m=+1134.022956241" watchObservedRunningTime="2026-10-30 18:52:00.155900338 +0000 UTC m=+1134.024318667"
Oct 30 18:52:01.121388 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.121348    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7" event={"ID":"3d2809d5-9d1e-4d21-baa5-9c38079d5afc","Type":"ContainerStarted","Data":"957a39163eebdcebafa97999d12be21155e9f2b94397ee1f33591d783c73454d"}
Oct 30 18:52:01.121843 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.121614    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"
Oct 30 18:52:01.150581 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.150524    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7" podStartSLOduration=3.150506278 podStartE2EDuration="3.150506278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:01.148413417 +0000 UTC m=+1135.016831738" watchObservedRunningTime="2026-10-30 18:52:01.150506278 +0000 UTC m=+1135.018924610"
Oct 30 18:52:01.328941 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.328912    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"]
Oct 30 18:52:01.333664 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.333639    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"
Oct 30 18:52:01.349204 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.348372    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ecef08\""
Oct 30 18:52:01.356961 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.356928    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"]
Oct 30 18:52:01.381496 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.381431    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"]
Oct 30 18:52:01.386647 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.386625    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"
Oct 30 18:52:01.400340 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.400313    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ecef08-dockercfg-nqzbz\""
Oct 30 18:52:01.402444 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.402422    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"]
Oct 30 18:52:01.453989 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.453953    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttflk\" (UniqueName: \"kubernetes.io/projected/ae55fd99-99c9-4917-aeeb-2f40a2b65a24-kube-api-access-ttflk\") pod \"payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w\" (UID: \"ae55fd99-99c9-4917-aeeb-2f40a2b65a24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"
Oct 30 18:52:01.454364 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.454337    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae55fd99-99c9-4917-aeeb-2f40a2b65a24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w\" (UID: \"ae55fd99-99c9-4917-aeeb-2f40a2b65a24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"
Oct 30 18:52:01.556474 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.556442    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttflk\" (UniqueName: \"kubernetes.io/projected/ae55fd99-99c9-4917-aeeb-2f40a2b65a24-kube-api-access-ttflk\") pod \"payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w\" (UID: \"ae55fd99-99c9-4917-aeeb-2f40a2b65a24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"
Oct 30 18:52:01.556644 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.556541    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae55fd99-99c9-4917-aeeb-2f40a2b65a24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w\" (UID: \"ae55fd99-99c9-4917-aeeb-2f40a2b65a24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"
Oct 30 18:52:01.556644 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.556580    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c1b2aa-d436-4ff2-a830-e09698aeecf2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp\" (UID: \"b4c1b2aa-d436-4ff2-a830-e09698aeecf2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"
Oct 30 18:52:01.556644 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.556609    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h6dn\" (UniqueName: \"kubernetes.io/projected/b4c1b2aa-d436-4ff2-a830-e09698aeecf2-kube-api-access-9h6dn\") pod \"payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp\" (UID: \"b4c1b2aa-d436-4ff2-a830-e09698aeecf2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"
Oct 30 18:52:01.557319 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.557290    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae55fd99-99c9-4917-aeeb-2f40a2b65a24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w\" (UID: \"ae55fd99-99c9-4917-aeeb-2f40a2b65a24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"
Oct 30 18:52:01.573785 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.573763    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttflk\" (UniqueName: \"kubernetes.io/projected/ae55fd99-99c9-4917-aeeb-2f40a2b65a24-kube-api-access-ttflk\") pod \"payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w\" (UID: \"ae55fd99-99c9-4917-aeeb-2f40a2b65a24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"
Oct 30 18:52:01.660197 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.658387    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c1b2aa-d436-4ff2-a830-e09698aeecf2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp\" (UID: \"b4c1b2aa-d436-4ff2-a830-e09698aeecf2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"
Oct 30 18:52:01.660197 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.658454    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h6dn\" (UniqueName: \"kubernetes.io/projected/b4c1b2aa-d436-4ff2-a830-e09698aeecf2-kube-api-access-9h6dn\") pod \"payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp\" (UID: \"b4c1b2aa-d436-4ff2-a830-e09698aeecf2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"
Oct 30 18:52:01.660197 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.658943    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"
Oct 30 18:52:01.660492 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.660390    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c1b2aa-d436-4ff2-a830-e09698aeecf2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp\" (UID: \"b4c1b2aa-d436-4ff2-a830-e09698aeecf2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"
Oct 30 18:52:01.722041 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:01.722006    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h6dn\" (UniqueName: \"kubernetes.io/projected/b4c1b2aa-d436-4ff2-a830-e09698aeecf2-kube-api-access-9h6dn\") pod \"payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp\" (UID: \"b4c1b2aa-d436-4ff2-a830-e09698aeecf2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"
Oct 30 18:52:02.025206 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:02.022418    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"
Oct 30 18:52:02.267556 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:52:02.267524    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae55fd99_99c9_4917_aeeb_2f40a2b65a24.slice/crio-f150996262d781a4092bf50e4ad4e534bfa84c13adfcce7e4e8f0b76c3aaf435 WatchSource:0}: Error finding container f150996262d781a4092bf50e4ad4e534bfa84c13adfcce7e4e8f0b76c3aaf435: Status 404 returned error can't find the container with id f150996262d781a4092bf50e4ad4e534bfa84c13adfcce7e4e8f0b76c3aaf435
Oct 30 18:52:02.269983 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:02.269959    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"]
Oct 30 18:52:02.353096 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:52:02.353068    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c1b2aa_d436_4ff2_a830_e09698aeecf2.slice/crio-7a76bebbfc3519bb95dd1be4f84d36aa815ffc15ddec8de1664944b7002b7dec WatchSource:0}: Error finding container 7a76bebbfc3519bb95dd1be4f84d36aa815ffc15ddec8de1664944b7002b7dec: Status 404 returned error can't find the container with id 7a76bebbfc3519bb95dd1be4f84d36aa815ffc15ddec8de1664944b7002b7dec
Oct 30 18:52:02.354644 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:02.354547    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"]
Oct 30 18:52:03.201332 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:03.201291    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp" event={"ID":"b4c1b2aa-d436-4ff2-a830-e09698aeecf2","Type":"ContainerStarted","Data":"53aabf1e60ee40795b8158c0fb03a7c992f1de1e48fb1b29dd8d48cf49b9278b"}
Oct 30 18:52:03.201520 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:03.201390    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp" event={"ID":"b4c1b2aa-d436-4ff2-a830-e09698aeecf2","Type":"ContainerStarted","Data":"7a76bebbfc3519bb95dd1be4f84d36aa815ffc15ddec8de1664944b7002b7dec"}
Oct 30 18:52:03.201520 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:03.201455    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"
Oct 30 18:52:03.204547 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:03.204517    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w" event={"ID":"ae55fd99-99c9-4917-aeeb-2f40a2b65a24","Type":"ContainerStarted","Data":"2d069eabea943fadccbdeaef8aeb64be7ccc389fe66382e9b2959f0692b4ce52"}
Oct 30 18:52:03.204695 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:03.204562    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w" event={"ID":"ae55fd99-99c9-4917-aeeb-2f40a2b65a24","Type":"ContainerStarted","Data":"f150996262d781a4092bf50e4ad4e534bfa84c13adfcce7e4e8f0b76c3aaf435"}
Oct 30 18:52:03.204695 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:03.204621    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"
Oct 30 18:52:03.227199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:03.227146    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp" podStartSLOduration=2.227112474 podStartE2EDuration="2.227112474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:03.223904693 +0000 UTC m=+1137.092323025" watchObservedRunningTime="2026-10-30 18:52:03.227112474 +0000 UTC m=+1137.095530805"
Oct 30 18:52:06.630262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:06.629211    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": read tcp 10.133.0.2:51220->10.133.0.101:15021: read: connection reset by peer" start-of-body=
Oct 30 18:52:06.630262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:06.629286    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h" podUID="5a4347ef-492a-4569-a344-1e683419ea7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": read tcp 10.133.0.2:51220->10.133.0.101:15021: read: connection reset by peer"
Oct 30 18:52:06.982985 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:06.982887    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w" podStartSLOduration=5.9828744369999995 podStartE2EDuration="5.982874437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:03.249230861 +0000 UTC m=+1137.117649194" watchObservedRunningTime="2026-10-30 18:52:06.982874437 +0000 UTC m=+1140.851292766"
Oct 30 18:52:06.983781 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:06.983763    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"]
Oct 30 18:52:06.987950 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:06.987935    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"
Oct 30 18:52:07.002315 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.002290    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"]
Oct 30 18:52:07.002792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.002775    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9ceefc\""
Oct 30 18:52:07.021044 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.021014    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9ceefc-dockercfg-ltm7z\""
Oct 30 18:52:07.129149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.124145    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6b55fc-3dab-4085-9f1c-672721d0538f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb\" (UID: \"ea6b55fc-3dab-4085-9f1c-672721d0538f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"
Oct 30 18:52:07.129149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.124301    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9d67\" (UniqueName: \"kubernetes.io/projected/ea6b55fc-3dab-4085-9f1c-672721d0538f-kube-api-access-f9d67\") pod \"payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb\" (UID: \"ea6b55fc-3dab-4085-9f1c-672721d0538f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"
Oct 30 18:52:07.233333 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.227330    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6b55fc-3dab-4085-9f1c-672721d0538f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb\" (UID: \"ea6b55fc-3dab-4085-9f1c-672721d0538f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"
Oct 30 18:52:07.233333 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.227516    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9d67\" (UniqueName: \"kubernetes.io/projected/ea6b55fc-3dab-4085-9f1c-672721d0538f-kube-api-access-f9d67\") pod \"payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb\" (UID: \"ea6b55fc-3dab-4085-9f1c-672721d0538f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"
Oct 30 18:52:07.233333 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.228825    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6b55fc-3dab-4085-9f1c-672721d0538f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb\" (UID: \"ea6b55fc-3dab-4085-9f1c-672721d0538f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"
Oct 30 18:52:07.366869 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.366839    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9d67\" (UniqueName: \"kubernetes.io/projected/ea6b55fc-3dab-4085-9f1c-672721d0538f-kube-api-access-f9d67\") pod \"payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb\" (UID: \"ea6b55fc-3dab-4085-9f1c-672721d0538f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"
Oct 30 18:52:07.380146 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.378066    4372 generic.go:356] "Generic (PLEG): container finished" podID="5a4347ef-492a-4569-a344-1e683419ea7f" containerID="d5a3a7c509e4450c64837712c5c3c33c35c3f210794aea437c3b46f44e2e9137" exitCode=0
Oct 30 18:52:07.380146 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.378257    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h" event={"ID":"5a4347ef-492a-4569-a344-1e683419ea7f","Type":"ContainerDied","Data":"d5a3a7c509e4450c64837712c5c3c33c35c3f210794aea437c3b46f44e2e9137"}
Oct 30 18:52:07.380146 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.378787    4372 scope.go:122] "RemoveContainer" containerID="d5a3a7c509e4450c64837712c5c3c33c35c3f210794aea437c3b46f44e2e9137"
Oct 30 18:52:07.382614 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.381077    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:07.382614 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.381196    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:07.382614 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.381306    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:07.602595 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.602562    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"
Oct 30 18:52:07.722993 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.722176    4372 scope.go:122] "RemoveContainer" containerID="09cfb2931ade9b998deb956293d9b9e7b36ea5a3be9028566de3723f726031b7"
Oct 30 18:52:07.738003 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.737977    4372 scope.go:122] "RemoveContainer" containerID="3748c3e0d855464e39edb46394504470c835cda942d04d090639e538ec9fb1d7"
Oct 30 18:52:07.752711 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.752604    4372 scope.go:122] "RemoveContainer" containerID="a4f80ce9e3c3c570b3ec56365ec61d52522f826327897b852ccf119f4903bb9b"
Oct 30 18:52:07.956179 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:07.955409    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"]
Oct 30 18:52:08.410770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:08.410676    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h" event={"ID":"5a4347ef-492a-4569-a344-1e683419ea7f","Type":"ContainerStarted","Data":"0174c36ba46f468afea05521fa725c6c9ca6559ff1ed5ca70e908fd3da0d8712"}
Oct 30 18:52:08.412511 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:08.412481    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb" event={"ID":"ea6b55fc-3dab-4085-9f1c-672721d0538f","Type":"ContainerStarted","Data":"61f314310bbdee6e621f86b1edf23fa4e35fbc36316301b60db74d4841f40178"}
Oct 30 18:52:08.412667 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:08.412527    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb" event={"ID":"ea6b55fc-3dab-4085-9f1c-672721d0538f","Type":"ContainerStarted","Data":"155d920d10c190ad537e8e4e13788c3c53937f670db570ff3bf1cc98e49035a4"}
Oct 30 18:52:08.412667 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:08.412548    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"
Oct 30 18:52:08.499551 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:08.499485    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb" podStartSLOduration=2.499460612 podStartE2EDuration="2.499460612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:08.466568434 +0000 UTC m=+1142.334986766" watchObservedRunningTime="2026-10-30 18:52:08.499460612 +0000 UTC m=+1142.367878943"
Oct 30 18:52:09.336932 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:09.336900    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:52:09.337503 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:09.337158    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:52:09.337503 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:09.337207    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h" podUID="5a4347ef-492a-4569-a344-1e683419ea7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 18:52:10.337941 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:10.337904    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:52:10.338360 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:10.337969    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h" podUID="5a4347ef-492a-4569-a344-1e683419ea7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 18:52:10.851772 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:10.851737    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"]
Oct 30 18:52:10.857237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:10.857209    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:10.862917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:10.862895    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ecef08-serving-cert\""
Oct 30 18:52:10.868637 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:10.868613    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"]
Oct 30 18:52:10.910401 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:10.910372    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-metrics-certs\") pod \"maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:10.910697 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:10.910636    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:10.910811 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:10.910715    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7w9\" (UniqueName: \"kubernetes.io/projected/bf1c2575-55bb-418a-8600-46e682bed18a-kube-api-access-qn7w9\") pod \"maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:10.910983 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:10.910966    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1c2575-55bb-418a-8600-46e682bed18a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:11.019035 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:11.015159    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:11.019035 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:11.015221    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7w9\" (UniqueName: \"kubernetes.io/projected/bf1c2575-55bb-418a-8600-46e682bed18a-kube-api-access-qn7w9\") pod \"maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:11.019035 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:11.015471    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1c2575-55bb-418a-8600-46e682bed18a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:11.019035 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:11.015591    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-metrics-certs\") pod \"maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:11.021208 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:11.021011    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1c2575-55bb-418a-8600-46e682bed18a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:11.021208 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:52:11.021161    4372 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-serving-cert: secret "maas-api-e2e-ipp-b-ecef08-serving-cert" not found
Oct 30 18:52:11.021417 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:52:11.021221    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-maas-api-tls podName:bf1c2575-55bb-418a-8600-46e682bed18a nodeName:}" failed. No retries permitted until 2026-10-30 18:52:11.521201824 +0000 UTC m=+1145.389620146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-maas-api-tls") pod "maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh" (UID: "bf1c2575-55bb-418a-8600-46e682bed18a") : secret "maas-api-e2e-ipp-b-ecef08-serving-cert" not found
Oct 30 18:52:11.060761 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:11.030183    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-metrics-certs\") pod \"maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:11.092156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:11.088055    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7w9\" (UniqueName: \"kubernetes.io/projected/bf1c2575-55bb-418a-8600-46e682bed18a-kube-api-access-qn7w9\") pod \"maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:11.148200 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:11.143368    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"
Oct 30 18:52:11.337974 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:11.337942    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:52:11.338407 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:11.337990    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h" podUID="5a4347ef-492a-4569-a344-1e683419ea7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 18:52:11.531726 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:11.531646    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:11.534311 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:11.534277    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:11.817199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:11.816843    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:12.149809 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:12.149774    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"
Oct 30 18:52:12.337912 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:12.337878    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:52:12.338116 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:12.337936    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h" podUID="5a4347ef-492a-4569-a344-1e683419ea7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 18:52:12.406701 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:12.406635    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"]
Oct 30 18:52:12.440157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:12.440095    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh" event={"ID":"bf1c2575-55bb-418a-8600-46e682bed18a","Type":"ContainerStarted","Data":"518cb2ef4efff5c893bc89950660bb7967193f643afad8af2c2bcda31a9d346b"}
Oct 30 18:52:13.344349 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:13.344305    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:52:13.344875 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:13.344397    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h" podUID="5a4347ef-492a-4569-a344-1e683419ea7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 18:52:13.449914 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:13.449878    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh" event={"ID":"bf1c2575-55bb-418a-8600-46e682bed18a","Type":"ContainerStarted","Data":"b848eea24c66522e873aac3b2b6477d7f52e589c0074d0a6fc8ab77ef14bf3e4"}
Oct 30 18:52:13.450120 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:13.450099    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:14.215833 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:14.215807    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"
Oct 30 18:52:14.216016 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:14.215872    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"
Oct 30 18:52:14.239199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:14.239150    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh" podStartSLOduration=3.809981676 podStartE2EDuration="4.239118114s" totalImagesPullingTime="429.136438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:13.50331529 +0000 UTC m=+1147.371733642" watchObservedRunningTime="2026-10-30 18:52:14.239118114 +0000 UTC m=+1148.107536445"
Oct 30 18:52:14.341440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:14.341411    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:52:14.341814 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:14.341791    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:52:14.342514 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:14.342496    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:52:17.806316 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:17.806285    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"]
Oct 30 18:52:17.811395 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:17.811375    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:17.835714 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:17.833614    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-9ceefc-serving-cert\""
Oct 30 18:52:17.841569 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:17.841545    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"]
Oct 30 18:52:17.907906 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:17.907864    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a17d135-89a9-4b2e-827a-f5a85b5048a3-metrics-certs\") pod \"maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:17.908069 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:17.907912    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnswg\" (UniqueName: \"kubernetes.io/projected/7a17d135-89a9-4b2e-827a-f5a85b5048a3-kube-api-access-pnswg\") pod \"maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:17.908332 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:17.908300    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a17d135-89a9-4b2e-827a-f5a85b5048a3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:17.908474 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:17.908455    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a17d135-89a9-4b2e-827a-f5a85b5048a3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:18.021619 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:18.010073    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a17d135-89a9-4b2e-827a-f5a85b5048a3-metrics-certs\") pod \"maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:18.021977 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:18.021947    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnswg\" (UniqueName: \"kubernetes.io/projected/7a17d135-89a9-4b2e-827a-f5a85b5048a3-kube-api-access-pnswg\") pod \"maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:18.022847 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:18.022736    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a17d135-89a9-4b2e-827a-f5a85b5048a3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:18.023946 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:18.023924    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a17d135-89a9-4b2e-827a-f5a85b5048a3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:18.028674 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:18.023742    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a17d135-89a9-4b2e-827a-f5a85b5048a3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:18.061260 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:18.061183    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a17d135-89a9-4b2e-827a-f5a85b5048a3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:18.063836 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:18.063755    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnswg\" (UniqueName: \"kubernetes.io/projected/7a17d135-89a9-4b2e-827a-f5a85b5048a3-kube-api-access-pnswg\") pod \"maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:18.066823 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:18.066788    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a17d135-89a9-4b2e-827a-f5a85b5048a3-metrics-certs\") pod \"maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:18.143503 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:18.143476    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:18.528867 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:18.527839    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"]
Oct 30 18:52:18.530667 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:18.530444    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs" event={"ID":"7a17d135-89a9-4b2e-827a-f5a85b5048a3","Type":"ContainerStarted","Data":"b44b05815f791ee9d7e9a8c5d0ecd509076a24f7095ce0d352ea25815f990d33"}
Oct 30 18:52:19.422302 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:19.422271    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"
Oct 30 18:52:19.463687 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:19.463657    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:52:19.542189 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:19.542159    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs" event={"ID":"7a17d135-89a9-4b2e-827a-f5a85b5048a3","Type":"ContainerStarted","Data":"16f65aea926e734ca830216f6872a0e7a451b91a7027bfebc8f3b43eb8884fd7"}
Oct 30 18:52:19.542449 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:19.542427    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:19.567751 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:19.567705    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs" podStartSLOduration=2.072995123 podStartE2EDuration="2.567692724s" totalImagesPullingTime="494.697601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:19.563185278 +0000 UTC m=+1153.431603609" watchObservedRunningTime="2026-10-30 18:52:19.567692724 +0000 UTC m=+1153.436111075"
Oct 30 18:52:22.624978 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:22.624930    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"]
Oct 30 18:52:22.634996 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:22.634963    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"]
Oct 30 18:52:25.593155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:25.593103    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:52:27.625775 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:27.625738    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7" podUID="3d2809d5-9d1e-4d21-baa5-9c38079d5afc" containerName="payload-processing" containerID="cri-o://957a39163eebdcebafa97999d12be21155e9f2b94397ee1f33591d783c73454d" gracePeriod=25
Oct 30 18:52:27.635468 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:27.635438    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx" podUID="58266961-626d-4032-a62d-1a080000767f" containerName="payload-pre-processing" containerID="cri-o://9fbfcf3a946b61f20ffb0b7290b741a6175177b78d3a77494384b396c2e06600" gracePeriod=25
Oct 30 18:52:27.695553 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:52:27.695522    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:52:28.032341 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.032314    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"
Oct 30 18:52:28.080361 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.080333    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2809d5-9d1e-4d21-baa5-9c38079d5afc-kube-api-access-9pdkp\" (UniqueName: \"kubernetes.io/projected/3d2809d5-9d1e-4d21-baa5-9c38079d5afc-kube-api-access-9pdkp\") pod \"3d2809d5-9d1e-4d21-baa5-9c38079d5afc\" (UID: \"3d2809d5-9d1e-4d21-baa5-9c38079d5afc\") "
Oct 30 18:52:28.080605 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.080587    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2809d5-9d1e-4d21-baa5-9c38079d5afc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2809d5-9d1e-4d21-baa5-9c38079d5afc-plugins-config-volume\") pod \"3d2809d5-9d1e-4d21-baa5-9c38079d5afc\" (UID: \"3d2809d5-9d1e-4d21-baa5-9c38079d5afc\") "
Oct 30 18:52:28.081212 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.081149    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2809d5-9d1e-4d21-baa5-9c38079d5afc-plugins-config-volume" pod "3d2809d5-9d1e-4d21-baa5-9c38079d5afc" (UID: "3d2809d5-9d1e-4d21-baa5-9c38079d5afc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:28.083047 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.083020    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2809d5-9d1e-4d21-baa5-9c38079d5afc-kube-api-access-9pdkp" pod "3d2809d5-9d1e-4d21-baa5-9c38079d5afc" (UID: "3d2809d5-9d1e-4d21-baa5-9c38079d5afc"). InnerVolumeSpecName "kube-api-access-9pdkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:28.087430 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.087411    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"
Oct 30 18:52:28.181911 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.181850    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58266961-626d-4032-a62d-1a080000767f-kube-api-access-zpjtf\" (UniqueName: \"kubernetes.io/projected/58266961-626d-4032-a62d-1a080000767f-kube-api-access-zpjtf\") pod \"58266961-626d-4032-a62d-1a080000767f\" (UID: \"58266961-626d-4032-a62d-1a080000767f\") "
Oct 30 18:52:28.182025 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.181987    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58266961-626d-4032-a62d-1a080000767f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58266961-626d-4032-a62d-1a080000767f-plugins-config-volume\") pod \"58266961-626d-4032-a62d-1a080000767f\" (UID: \"58266961-626d-4032-a62d-1a080000767f\") "
Oct 30 18:52:28.182325 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.182304    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58266961-626d-4032-a62d-1a080000767f-plugins-config-volume" pod "58266961-626d-4032-a62d-1a080000767f" (UID: "58266961-626d-4032-a62d-1a080000767f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:28.182609 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.182594    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pdkp\" (UniqueName: \"kubernetes.io/projected/3d2809d5-9d1e-4d21-baa5-9c38079d5afc-kube-api-access-9pdkp\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:28.182661 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.182612    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58266961-626d-4032-a62d-1a080000767f-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:28.182661 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.182621    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2809d5-9d1e-4d21-baa5-9c38079d5afc-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:28.184114 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.184094    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58266961-626d-4032-a62d-1a080000767f-kube-api-access-zpjtf" pod "58266961-626d-4032-a62d-1a080000767f" (UID: "58266961-626d-4032-a62d-1a080000767f"). InnerVolumeSpecName "kube-api-access-zpjtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:28.283943 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.283911    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpjtf\" (UniqueName: \"kubernetes.io/projected/58266961-626d-4032-a62d-1a080000767f-kube-api-access-zpjtf\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:28.380553 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.380519    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:52:28.689159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.680772    4372 generic.go:356] "Generic (PLEG): container finished" podID="3d2809d5-9d1e-4d21-baa5-9c38079d5afc" containerID="957a39163eebdcebafa97999d12be21155e9f2b94397ee1f33591d783c73454d" exitCode=0
Oct 30 18:52:28.689159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.681073    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7" event={"ID":"3d2809d5-9d1e-4d21-baa5-9c38079d5afc","Type":"ContainerDied","Data":"957a39163eebdcebafa97999d12be21155e9f2b94397ee1f33591d783c73454d"}
Oct 30 18:52:28.689159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.681155    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7" event={"ID":"3d2809d5-9d1e-4d21-baa5-9c38079d5afc","Type":"ContainerDied","Data":"e2d7ea0897a48aa6a25d4a7330d8e1956eaef377ac4c21cfd6a6c5f7c8d0df3e"}
Oct 30 18:52:28.689159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.681188    4372 scope.go:122] "RemoveContainer" containerID="957a39163eebdcebafa97999d12be21155e9f2b94397ee1f33591d783c73454d"
Oct 30 18:52:28.689159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.681253    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"
Oct 30 18:52:28.715223 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.715079    4372 generic.go:356] "Generic (PLEG): container finished" podID="58266961-626d-4032-a62d-1a080000767f" containerID="9fbfcf3a946b61f20ffb0b7290b741a6175177b78d3a77494384b396c2e06600" exitCode=0
Oct 30 18:52:28.715401 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.715243    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx" event={"ID":"58266961-626d-4032-a62d-1a080000767f","Type":"ContainerDied","Data":"9fbfcf3a946b61f20ffb0b7290b741a6175177b78d3a77494384b396c2e06600"}
Oct 30 18:52:28.715401 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.715356    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx" event={"ID":"58266961-626d-4032-a62d-1a080000767f","Type":"ContainerDied","Data":"1021a7409aec75ec24eb39988ffc78e5de144bf66484b72a31877109b1874675"}
Oct 30 18:52:28.715523 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.715475    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"
Oct 30 18:52:28.760754 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.760713    4372 scope.go:122] "RemoveContainer" containerID="957a39163eebdcebafa97999d12be21155e9f2b94397ee1f33591d783c73454d"
Oct 30 18:52:28.761854 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:52:28.761820    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"957a39163eebdcebafa97999d12be21155e9f2b94397ee1f33591d783c73454d\": container with ID starting with 957a39163eebdcebafa97999d12be21155e9f2b94397ee1f33591d783c73454d not found: ID does not exist" containerID="957a39163eebdcebafa97999d12be21155e9f2b94397ee1f33591d783c73454d"
Oct 30 18:52:28.761977 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.761866    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957a39163eebdcebafa97999d12be21155e9f2b94397ee1f33591d783c73454d"} err="failed to get container status \"957a39163eebdcebafa97999d12be21155e9f2b94397ee1f33591d783c73454d\": rpc error: code = NotFound desc = could not find container \"957a39163eebdcebafa97999d12be21155e9f2b94397ee1f33591d783c73454d\": container with ID starting with 957a39163eebdcebafa97999d12be21155e9f2b94397ee1f33591d783c73454d not found: ID does not exist"
Oct 30 18:52:28.761977 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.761892    4372 scope.go:122] "RemoveContainer" containerID="9fbfcf3a946b61f20ffb0b7290b741a6175177b78d3a77494384b396c2e06600"
Oct 30 18:52:28.823672 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.818202    4372 scope.go:122] "RemoveContainer" containerID="9fbfcf3a946b61f20ffb0b7290b741a6175177b78d3a77494384b396c2e06600"
Oct 30 18:52:28.828105 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:52:28.824036    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fbfcf3a946b61f20ffb0b7290b741a6175177b78d3a77494384b396c2e06600\": container with ID starting with 9fbfcf3a946b61f20ffb0b7290b741a6175177b78d3a77494384b396c2e06600 not found: ID does not exist" containerID="9fbfcf3a946b61f20ffb0b7290b741a6175177b78d3a77494384b396c2e06600"
Oct 30 18:52:28.828105 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.824079    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fbfcf3a946b61f20ffb0b7290b741a6175177b78d3a77494384b396c2e06600"} err="failed to get container status \"9fbfcf3a946b61f20ffb0b7290b741a6175177b78d3a77494384b396c2e06600\": rpc error: code = NotFound desc = could not find container \"9fbfcf3a946b61f20ffb0b7290b741a6175177b78d3a77494384b396c2e06600\": container with ID starting with 9fbfcf3a946b61f20ffb0b7290b741a6175177b78d3a77494384b396c2e06600 not found: ID does not exist"
Oct 30 18:52:28.858646 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.858265    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"]
Oct 30 18:52:28.891145 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.888214    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fd4ed-5d4fc874bd-r6wb7"]
Oct 30 18:52:28.904214 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.901252    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"]
Oct 30 18:52:28.927629 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:28.927285    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fd4ed-567d6dfc75-7z7cx"]
Oct 30 18:52:29.935676 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:29.935645    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"]
Oct 30 18:52:29.957093 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:29.957065    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"]
Oct 30 18:52:30.267080 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:30.267003    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2809d5-9d1e-4d21-baa5-9c38079d5afc" path="/var/lib/kubelet/pods/3d2809d5-9d1e-4d21-baa5-9c38079d5afc/volumes"
Oct 30 18:52:30.267558 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:30.267544    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58266961-626d-4032-a62d-1a080000767f" path="/var/lib/kubelet/pods/58266961-626d-4032-a62d-1a080000767f/volumes"
Oct 30 18:52:34.951156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:34.949357    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp" podUID="b4c1b2aa-d436-4ff2-a830-e09698aeecf2" containerName="payload-processing" containerID="cri-o://53aabf1e60ee40795b8158c0fb03a7c992f1de1e48fb1b29dd8d48cf49b9278b" gracePeriod=25
Oct 30 18:52:34.961007 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:34.960427    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w" podUID="ae55fd99-99c9-4917-aeeb-2f40a2b65a24" containerName="payload-pre-processing" containerID="cri-o://2d069eabea943fadccbdeaef8aeb64be7ccc389fe66382e9b2959f0692b4ce52" gracePeriod=25
Oct 30 18:52:35.053485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:35.053452    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"]
Oct 30 18:52:35.053923 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:35.053889    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf" podUID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerName="istio-proxy" containerID="cri-o://6526dc920598f47b104a5557e7057e2e2d76dfa471adc8f8057a460484a387cb" gracePeriod=30
Oct 30 18:52:36.157022 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.156989    4372 generic.go:356] "Generic (PLEG): container finished" podID="b4c1b2aa-d436-4ff2-a830-e09698aeecf2" containerID="53aabf1e60ee40795b8158c0fb03a7c992f1de1e48fb1b29dd8d48cf49b9278b" exitCode=0
Oct 30 18:52:36.157768 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.157738    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp" event={"ID":"b4c1b2aa-d436-4ff2-a830-e09698aeecf2","Type":"ContainerDied","Data":"53aabf1e60ee40795b8158c0fb03a7c992f1de1e48fb1b29dd8d48cf49b9278b"}
Oct 30 18:52:36.160477 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.160451    4372 generic.go:356] "Generic (PLEG): container finished" podID="ae55fd99-99c9-4917-aeeb-2f40a2b65a24" containerID="2d069eabea943fadccbdeaef8aeb64be7ccc389fe66382e9b2959f0692b4ce52" exitCode=0
Oct 30 18:52:36.160683 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.160665    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w" event={"ID":"ae55fd99-99c9-4917-aeeb-2f40a2b65a24","Type":"ContainerDied","Data":"2d069eabea943fadccbdeaef8aeb64be7ccc389fe66382e9b2959f0692b4ce52"}
Oct 30 18:52:36.185115 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.185089    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"
Oct 30 18:52:36.243207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.233783    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"]
Oct 30 18:52:36.266159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.265698    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"
Oct 30 18:52:36.333149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.332934    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c1b2aa-d436-4ff2-a830-e09698aeecf2-kube-api-access-9h6dn\" (UniqueName: \"kubernetes.io/projected/b4c1b2aa-d436-4ff2-a830-e09698aeecf2-kube-api-access-9h6dn\") pod \"b4c1b2aa-d436-4ff2-a830-e09698aeecf2\" (UID: \"b4c1b2aa-d436-4ff2-a830-e09698aeecf2\") "
Oct 30 18:52:36.333149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.333041    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4c1b2aa-d436-4ff2-a830-e09698aeecf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c1b2aa-d436-4ff2-a830-e09698aeecf2-plugins-config-volume\") pod \"b4c1b2aa-d436-4ff2-a830-e09698aeecf2\" (UID: \"b4c1b2aa-d436-4ff2-a830-e09698aeecf2\") "
Oct 30 18:52:36.339147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.333553    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c1b2aa-d436-4ff2-a830-e09698aeecf2-plugins-config-volume" pod "b4c1b2aa-d436-4ff2-a830-e09698aeecf2" (UID: "b4c1b2aa-d436-4ff2-a830-e09698aeecf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:36.339147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.335818    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c1b2aa-d436-4ff2-a830-e09698aeecf2-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:36.343191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.341880    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c1b2aa-d436-4ff2-a830-e09698aeecf2-kube-api-access-9h6dn" pod "b4c1b2aa-d436-4ff2-a830-e09698aeecf2" (UID: "b4c1b2aa-d436-4ff2-a830-e09698aeecf2"). InnerVolumeSpecName "kube-api-access-9h6dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:36.456164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.453704    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae55fd99-99c9-4917-aeeb-2f40a2b65a24-kube-api-access-ttflk\" (UniqueName: \"kubernetes.io/projected/ae55fd99-99c9-4917-aeeb-2f40a2b65a24-kube-api-access-ttflk\") pod \"ae55fd99-99c9-4917-aeeb-2f40a2b65a24\" (UID: \"ae55fd99-99c9-4917-aeeb-2f40a2b65a24\") "
Oct 30 18:52:36.456164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.453981    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae55fd99-99c9-4917-aeeb-2f40a2b65a24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae55fd99-99c9-4917-aeeb-2f40a2b65a24-plugins-config-volume\") pod \"ae55fd99-99c9-4917-aeeb-2f40a2b65a24\" (UID: \"ae55fd99-99c9-4917-aeeb-2f40a2b65a24\") "
Oct 30 18:52:36.456164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.454904    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h6dn\" (UniqueName: \"kubernetes.io/projected/b4c1b2aa-d436-4ff2-a830-e09698aeecf2-kube-api-access-9h6dn\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:36.460157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.457199    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae55fd99-99c9-4917-aeeb-2f40a2b65a24-plugins-config-volume" pod "ae55fd99-99c9-4917-aeeb-2f40a2b65a24" (UID: "ae55fd99-99c9-4917-aeeb-2f40a2b65a24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:36.484213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.481403    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae55fd99-99c9-4917-aeeb-2f40a2b65a24-kube-api-access-ttflk" pod "ae55fd99-99c9-4917-aeeb-2f40a2b65a24" (UID: "ae55fd99-99c9-4917-aeeb-2f40a2b65a24"). InnerVolumeSpecName "kube-api-access-ttflk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:36.600203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.590192    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae55fd99-99c9-4917-aeeb-2f40a2b65a24-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:36.600203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:36.590245    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttflk\" (UniqueName: \"kubernetes.io/projected/ae55fd99-99c9-4917-aeeb-2f40a2b65a24-kube-api-access-ttflk\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:37.325202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:37.325156    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp" event={"ID":"b4c1b2aa-d436-4ff2-a830-e09698aeecf2","Type":"ContainerDied","Data":"7a76bebbfc3519bb95dd1be4f84d36aa815ffc15ddec8de1664944b7002b7dec"}
Oct 30 18:52:37.325722 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:37.325229    4372 scope.go:122] "RemoveContainer" containerID="53aabf1e60ee40795b8158c0fb03a7c992f1de1e48fb1b29dd8d48cf49b9278b"
Oct 30 18:52:37.325722 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:37.325332    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"
Oct 30 18:52:37.333185 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:37.333118    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"
Oct 30 18:52:37.333305 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:37.333160    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w" event={"ID":"ae55fd99-99c9-4917-aeeb-2f40a2b65a24","Type":"ContainerDied","Data":"f150996262d781a4092bf50e4ad4e534bfa84c13adfcce7e4e8f0b76c3aaf435"}
Oct 30 18:52:37.344638 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:37.344621    4372 scope.go:122] "RemoveContainer" containerID="2d069eabea943fadccbdeaef8aeb64be7ccc389fe66382e9b2959f0692b4ce52"
Oct 30 18:52:37.406748 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:37.406715    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"]
Oct 30 18:52:37.414359 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:37.414333    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecef08-78b8574dd4-ppxcp"]
Oct 30 18:52:37.435911 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:37.435881    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"]
Oct 30 18:52:37.452635 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:37.452608    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-6df68f64d7-m2x2w"]
Oct 30 18:52:38.268817 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:38.268789    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae55fd99-99c9-4917-aeeb-2f40a2b65a24" path="/var/lib/kubelet/pods/ae55fd99-99c9-4917-aeeb-2f40a2b65a24/volumes"
Oct 30 18:52:38.269272 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:38.269258    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c1b2aa-d436-4ff2-a830-e09698aeecf2" path="/var/lib/kubelet/pods/b4c1b2aa-d436-4ff2-a830-e09698aeecf2/volumes"
Oct 30 18:52:39.066163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.012980    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"]
Oct 30 18:52:39.088149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.067022    4372 cpu_manager.go:414] "containerMap: removing container" podUID="3d2809d5-9d1e-4d21-baa5-9c38079d5afc" containerName="payload-processing"
Oct 30 18:52:39.126152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.103187    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2809d5-9d1e-4d21-baa5-9c38079d5afc" containerName="payload-processing"
Oct 30 18:52:39.126152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.103233    4372 cpu_manager.go:414] "containerMap: removing container" podUID="ae55fd99-99c9-4917-aeeb-2f40a2b65a24" containerName="payload-pre-processing"
Oct 30 18:52:39.126152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.103254    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae55fd99-99c9-4917-aeeb-2f40a2b65a24" containerName="payload-pre-processing"
Oct 30 18:52:39.126152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.103340    4372 cpu_manager.go:414] "containerMap: removing container" podUID="58266961-626d-4032-a62d-1a080000767f" containerName="payload-pre-processing"
Oct 30 18:52:39.126152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.103347    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58266961-626d-4032-a62d-1a080000767f" containerName="payload-pre-processing"
Oct 30 18:52:39.126152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.103374    4372 cpu_manager.go:414] "containerMap: removing container" podUID="b4c1b2aa-d436-4ff2-a830-e09698aeecf2" containerName="payload-processing"
Oct 30 18:52:39.126152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.103381    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c1b2aa-d436-4ff2-a830-e09698aeecf2" containerName="payload-processing"
Oct 30 18:52:39.126152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.103608    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="58266961-626d-4032-a62d-1a080000767f" containerName="payload-pre-processing"
Oct 30 18:52:39.126152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.103631    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c1b2aa-d436-4ff2-a830-e09698aeecf2" containerName="payload-processing"
Oct 30 18:52:39.126152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.103649    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2809d5-9d1e-4d21-baa5-9c38079d5afc" containerName="payload-processing"
Oct 30 18:52:39.126152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.103661    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae55fd99-99c9-4917-aeeb-2f40a2b65a24" containerName="payload-pre-processing"
Oct 30 18:52:39.293155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.289619    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"
Oct 30 18:52:39.316218 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.300445    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ecef08\""
Oct 30 18:52:39.316218 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.302210    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ecef08-dockercfg-87gbb\""
Oct 30 18:52:39.325308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.325277    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"]
Oct 30 18:52:39.542742 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.542693    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6009aa5-288e-4eba-903a-7c4ad6a93c44-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs\" (UID: \"d6009aa5-288e-4eba-903a-7c4ad6a93c44\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"
Oct 30 18:52:39.542942 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.542757    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j64dm\" (UniqueName: \"kubernetes.io/projected/d6009aa5-288e-4eba-903a-7c4ad6a93c44-kube-api-access-j64dm\") pod \"payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs\" (UID: \"d6009aa5-288e-4eba-903a-7c4ad6a93c44\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"
Oct 30 18:52:39.644687 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.644646    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6009aa5-288e-4eba-903a-7c4ad6a93c44-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs\" (UID: \"d6009aa5-288e-4eba-903a-7c4ad6a93c44\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"
Oct 30 18:52:39.644687 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.644690    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j64dm\" (UniqueName: \"kubernetes.io/projected/d6009aa5-288e-4eba-903a-7c4ad6a93c44-kube-api-access-j64dm\") pod \"payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs\" (UID: \"d6009aa5-288e-4eba-903a-7c4ad6a93c44\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"
Oct 30 18:52:39.645547 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.645527    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6009aa5-288e-4eba-903a-7c4ad6a93c44-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs\" (UID: \"d6009aa5-288e-4eba-903a-7c4ad6a93c44\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"
Oct 30 18:52:39.658104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.658067    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j64dm\" (UniqueName: \"kubernetes.io/projected/d6009aa5-288e-4eba-903a-7c4ad6a93c44-kube-api-access-j64dm\") pod \"payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs\" (UID: \"d6009aa5-288e-4eba-903a-7c4ad6a93c44\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"
Oct 30 18:52:39.802414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.802364    4372 patch_prober.go:29] interesting pod/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:52:39.802627 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.802455    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf" podUID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:52:39.940054 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:39.939966    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"
Oct 30 18:52:40.247421 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:52:40.247380    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6009aa5_288e_4eba_903a_7c4ad6a93c44.slice/crio-11f7356d0bab93dacbb73fbbc25b71c0956b3a487ebf2b720a41166cabf189e0 WatchSource:0}: Error finding container 11f7356d0bab93dacbb73fbbc25b71c0956b3a487ebf2b720a41166cabf189e0: Status 404 returned error can't find the container with id 11f7356d0bab93dacbb73fbbc25b71c0956b3a487ebf2b720a41166cabf189e0
Oct 30 18:52:40.248706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.248589    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"]
Oct 30 18:52:40.379150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.377717    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs" event={"ID":"d6009aa5-288e-4eba-903a-7c4ad6a93c44","Type":"ContainerStarted","Data":"11f7356d0bab93dacbb73fbbc25b71c0956b3a487ebf2b720a41166cabf189e0"}
Oct 30 18:52:40.383415 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.383061    4372 generic.go:356] "Generic (PLEG): container finished" podID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerID="6526dc920598f47b104a5557e7057e2e2d76dfa471adc8f8057a460484a387cb" exitCode=0
Oct 30 18:52:40.383415 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.383247    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf" event={"ID":"caf281ca-692b-48c8-84d7-13ca8e6e7489","Type":"ContainerDied","Data":"6526dc920598f47b104a5557e7057e2e2d76dfa471adc8f8057a460484a387cb"}
Oct 30 18:52:40.383415 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.383302    4372 scope.go:122] "RemoveContainer" containerID="903c5942727be773f1336a3b5866d3e00b05992470e4bc2466b3f43868ef57af"
Oct 30 18:52:40.499418 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.499388    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:52:40.667194 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.667156    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-workload-socket\") pod \"caf281ca-692b-48c8-84d7-13ca8e6e7489\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") "
Oct 30 18:52:40.667381 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.667206    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-token\" (UniqueName: \"kubernetes.io/projected/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-token\") pod \"caf281ca-692b-48c8-84d7-13ca8e6e7489\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") "
Oct 30 18:52:40.667381 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.667264    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf281ca-692b-48c8-84d7-13ca8e6e7489-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf281ca-692b-48c8-84d7-13ca8e6e7489-istiod-ca-cert\") pod \"caf281ca-692b-48c8-84d7-13ca8e6e7489\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") "
Oct 30 18:52:40.667827 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.667400    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-envoy\") pod \"caf281ca-692b-48c8-84d7-13ca8e6e7489\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") "
Oct 30 18:52:40.667827 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.667418    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-workload-socket" pod "caf281ca-692b-48c8-84d7-13ca8e6e7489" (UID: "caf281ca-692b-48c8-84d7-13ca8e6e7489"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:40.667827 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.667457    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-credential-socket\") pod \"caf281ca-692b-48c8-84d7-13ca8e6e7489\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") "
Oct 30 18:52:40.667827 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.667490    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-podinfo\") pod \"caf281ca-692b-48c8-84d7-13ca8e6e7489\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") "
Oct 30 18:52:40.667827 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.667604    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf281ca-692b-48c8-84d7-13ca8e6e7489-kube-api-access-ctsqb\" (UniqueName: \"kubernetes.io/projected/caf281ca-692b-48c8-84d7-13ca8e6e7489-kube-api-access-ctsqb\") pod \"caf281ca-692b-48c8-84d7-13ca8e6e7489\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") "
Oct 30 18:52:40.667827 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.667653    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-workload-certs\") pod \"caf281ca-692b-48c8-84d7-13ca8e6e7489\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") "
Oct 30 18:52:40.667827 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.667722    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-data\") pod \"caf281ca-692b-48c8-84d7-13ca8e6e7489\" (UID: \"caf281ca-692b-48c8-84d7-13ca8e6e7489\") "
Oct 30 18:52:40.668338 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.667929    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf281ca-692b-48c8-84d7-13ca8e6e7489-istiod-ca-cert" pod "caf281ca-692b-48c8-84d7-13ca8e6e7489" (UID: "caf281ca-692b-48c8-84d7-13ca8e6e7489"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:40.668668 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.668471    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:40.668668 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.668496    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf281ca-692b-48c8-84d7-13ca8e6e7489-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:40.669227 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.669202    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-credential-socket" pod "caf281ca-692b-48c8-84d7-13ca8e6e7489" (UID: "caf281ca-692b-48c8-84d7-13ca8e6e7489"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:40.669323 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.669299    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-workload-certs" pod "caf281ca-692b-48c8-84d7-13ca8e6e7489" (UID: "caf281ca-692b-48c8-84d7-13ca8e6e7489"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:40.669396 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.669356    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-data" pod "caf281ca-692b-48c8-84d7-13ca8e6e7489" (UID: "caf281ca-692b-48c8-84d7-13ca8e6e7489"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:40.670560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.670531    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-envoy" pod "caf281ca-692b-48c8-84d7-13ca8e6e7489" (UID: "caf281ca-692b-48c8-84d7-13ca8e6e7489"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:40.671474 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.671451    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-podinfo" pod "caf281ca-692b-48c8-84d7-13ca8e6e7489" (UID: "caf281ca-692b-48c8-84d7-13ca8e6e7489"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:52:40.671720 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.671698    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf281ca-692b-48c8-84d7-13ca8e6e7489-kube-api-access-ctsqb" pod "caf281ca-692b-48c8-84d7-13ca8e6e7489" (UID: "caf281ca-692b-48c8-84d7-13ca8e6e7489"). InnerVolumeSpecName "kube-api-access-ctsqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:40.672157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.672113    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-token" pod "caf281ca-692b-48c8-84d7-13ca8e6e7489" (UID: "caf281ca-692b-48c8-84d7-13ca8e6e7489"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:40.770066 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.769995    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:40.770066 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.770020    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:40.770066 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.770029    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:40.770066 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.770040    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:40.770066 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.770048    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf281ca-692b-48c8-84d7-13ca8e6e7489-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:40.770066 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.770058    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctsqb\" (UniqueName: \"kubernetes.io/projected/caf281ca-692b-48c8-84d7-13ca8e6e7489-kube-api-access-ctsqb\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:40.770066 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:40.770066    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf281ca-692b-48c8-84d7-13ca8e6e7489-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:41.236757 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:41.234203    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb" podUID="ea6b55fc-3dab-4085-9f1c-672721d0538f" containerName="payload-pre-processing" containerID="cri-o://61f314310bbdee6e621f86b1edf23fa4e35fbc36316301b60db74d4841f40178" gracePeriod=25
Oct 30 18:52:41.397159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:41.394370    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf" event={"ID":"caf281ca-692b-48c8-84d7-13ca8e6e7489","Type":"ContainerDied","Data":"5226bd71780cbcbb8bb6fb9634ca9d5f2663b6935b7933f628d225cdeca9490b"}
Oct 30 18:52:41.397159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:41.394445    4372 scope.go:122] "RemoveContainer" containerID="6526dc920598f47b104a5557e7057e2e2d76dfa471adc8f8057a460484a387cb"
Oct 30 18:52:41.397159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:41.394552    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"
Oct 30 18:52:41.406977 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:41.406939    4372 generic.go:356] "Generic (PLEG): container finished" podID="ea6b55fc-3dab-4085-9f1c-672721d0538f" containerID="61f314310bbdee6e621f86b1edf23fa4e35fbc36316301b60db74d4841f40178" exitCode=0
Oct 30 18:52:41.407220 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:41.407083    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb" event={"ID":"ea6b55fc-3dab-4085-9f1c-672721d0538f","Type":"ContainerDied","Data":"61f314310bbdee6e621f86b1edf23fa4e35fbc36316301b60db74d4841f40178"}
Oct 30 18:52:41.450451 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:41.450415    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"]
Oct 30 18:52:41.458173 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:41.458145    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2aada387-openshift-default-6dbbdcd6f-4tcwf"]
Oct 30 18:52:41.647189 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:41.647152    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"
Oct 30 18:52:41.686859 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:41.686820    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6b55fc-3dab-4085-9f1c-672721d0538f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6b55fc-3dab-4085-9f1c-672721d0538f-plugins-config-volume\") pod \"ea6b55fc-3dab-4085-9f1c-672721d0538f\" (UID: \"ea6b55fc-3dab-4085-9f1c-672721d0538f\") "
Oct 30 18:52:41.687031 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:41.686916    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6b55fc-3dab-4085-9f1c-672721d0538f-kube-api-access-f9d67\" (UniqueName: \"kubernetes.io/projected/ea6b55fc-3dab-4085-9f1c-672721d0538f-kube-api-access-f9d67\") pod \"ea6b55fc-3dab-4085-9f1c-672721d0538f\" (UID: \"ea6b55fc-3dab-4085-9f1c-672721d0538f\") "
Oct 30 18:52:41.687307 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:41.687275    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6b55fc-3dab-4085-9f1c-672721d0538f-plugins-config-volume" pod "ea6b55fc-3dab-4085-9f1c-672721d0538f" (UID: "ea6b55fc-3dab-4085-9f1c-672721d0538f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:41.687782 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:41.687759    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6b55fc-3dab-4085-9f1c-672721d0538f-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:41.701370 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:41.701323    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6b55fc-3dab-4085-9f1c-672721d0538f-kube-api-access-f9d67" pod "ea6b55fc-3dab-4085-9f1c-672721d0538f" (UID: "ea6b55fc-3dab-4085-9f1c-672721d0538f"). InnerVolumeSpecName "kube-api-access-f9d67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:41.789531 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:41.789447    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9d67\" (UniqueName: \"kubernetes.io/projected/ea6b55fc-3dab-4085-9f1c-672721d0538f-kube-api-access-f9d67\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:42.015321 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:42.015271    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:52:42.017404 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:42.015360    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:42.284026 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:42.283994    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf281ca-692b-48c8-84d7-13ca8e6e7489" path="/var/lib/kubelet/pods/caf281ca-692b-48c8-84d7-13ca8e6e7489/volumes"
Oct 30 18:52:42.417981 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:42.417940    4372 generic.go:356] "Generic (PLEG): container finished" podID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" containerID="0fb14a8ccf447af93953cf2b5ac08e30bb33616ce4b9d0e69f21b9e723567e34" exitCode=0
Oct 30 18:52:42.418625 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:42.417995    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc" event={"ID":"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04","Type":"ContainerDied","Data":"0fb14a8ccf447af93953cf2b5ac08e30bb33616ce4b9d0e69f21b9e723567e34"}
Oct 30 18:52:42.419179 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:42.419157    4372 scope.go:122] "RemoveContainer" containerID="0fb14a8ccf447af93953cf2b5ac08e30bb33616ce4b9d0e69f21b9e723567e34"
Oct 30 18:52:42.420681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:42.420655    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb" event={"ID":"ea6b55fc-3dab-4085-9f1c-672721d0538f","Type":"ContainerDied","Data":"155d920d10c190ad537e8e4e13788c3c53937f670db570ff3bf1cc98e49035a4"}
Oct 30 18:52:42.420789 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:42.420723    4372 scope.go:122] "RemoveContainer" containerID="61f314310bbdee6e621f86b1edf23fa4e35fbc36316301b60db74d4841f40178"
Oct 30 18:52:42.420909 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:42.420891    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"
Oct 30 18:52:42.421323 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:42.421271    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:42.421421 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:42.421361    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:42.421421 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:42.421404    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:42.470931 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:42.470892    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"]
Oct 30 18:52:42.477596 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:42.477570    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ceefc-6b5585b8b7-djkzb"]
Oct 30 18:52:43.014884 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.014807    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:52:43.050119 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.050029    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"]
Oct 30 18:52:43.051436 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.051396    4372 cpu_manager.go:414] "containerMap: removing container" podUID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerName="istio-proxy"
Oct 30 18:52:43.051637 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.051620    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerName="istio-proxy"
Oct 30 18:52:43.051751 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.051739    4372 cpu_manager.go:414] "containerMap: removing container" podUID="ea6b55fc-3dab-4085-9f1c-672721d0538f" containerName="payload-pre-processing"
Oct 30 18:52:43.051838 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.051824    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6b55fc-3dab-4085-9f1c-672721d0538f" containerName="payload-pre-processing"
Oct 30 18:52:43.052098 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.052085    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerName="istio-proxy"
Oct 30 18:52:43.052250 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.052237    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6b55fc-3dab-4085-9f1c-672721d0538f" containerName="payload-pre-processing"
Oct 30 18:52:43.061282 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.060421    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"
Oct 30 18:52:43.065576 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.065549    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-45f649-dockercfg-kntnk\""
Oct 30 18:52:43.065724 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.065653    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-45f649\""
Oct 30 18:52:43.068590 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.068564    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"]
Oct 30 18:52:43.158249 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.158207    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75dn2\" (UniqueName: \"kubernetes.io/projected/6f837717-c395-406e-af47-b93f2213a2bb-kube-api-access-75dn2\") pod \"payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr\" (UID: \"6f837717-c395-406e-af47-b93f2213a2bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"
Oct 30 18:52:43.158744 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.158723    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f837717-c395-406e-af47-b93f2213a2bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr\" (UID: \"6f837717-c395-406e-af47-b93f2213a2bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"
Oct 30 18:52:43.285276 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.285239    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f837717-c395-406e-af47-b93f2213a2bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr\" (UID: \"6f837717-c395-406e-af47-b93f2213a2bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"
Oct 30 18:52:43.289151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.285760    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75dn2\" (UniqueName: \"kubernetes.io/projected/6f837717-c395-406e-af47-b93f2213a2bb-kube-api-access-75dn2\") pod \"payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr\" (UID: \"6f837717-c395-406e-af47-b93f2213a2bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"
Oct 30 18:52:43.338206 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.323461    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f837717-c395-406e-af47-b93f2213a2bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr\" (UID: \"6f837717-c395-406e-af47-b93f2213a2bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"
Oct 30 18:52:43.451170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.451117    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75dn2\" (UniqueName: \"kubernetes.io/projected/6f837717-c395-406e-af47-b93f2213a2bb-kube-api-access-75dn2\") pod \"payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr\" (UID: \"6f837717-c395-406e-af47-b93f2213a2bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"
Oct 30 18:52:43.545925 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.545882    4372 generic.go:356] "Generic (PLEG): container finished" podID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerID="2adf405b22604db675e0895e8d6dd795a17d4de6976b9902cef1ed0ec9bf3d2e" exitCode=0
Oct 30 18:52:43.549613 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.549577    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" event={"ID":"2313cd49-c455-47e7-94df-091ce8c2edf9","Type":"ContainerDied","Data":"2adf405b22604db675e0895e8d6dd795a17d4de6976b9902cef1ed0ec9bf3d2e"}
Oct 30 18:52:43.550539 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.550507    4372 scope.go:122] "RemoveContainer" containerID="2adf405b22604db675e0895e8d6dd795a17d4de6976b9902cef1ed0ec9bf3d2e"
Oct 30 18:52:43.553719 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.553667    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:43.554152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.553922    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:43.554152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.554000    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:52:43.555009 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.554952    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc" event={"ID":"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04","Type":"ContainerStarted","Data":"cf11e1148fee1d90cf87dd839e5ded74671015c76bc6e4036a1b3fa8ef543877"}
Oct 30 18:52:43.560064 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.559369    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs" event={"ID":"d6009aa5-288e-4eba-903a-7c4ad6a93c44","Type":"ContainerStarted","Data":"cda298eededcb070261bf1d5828ebe0a612a991a0725879a620da48d5ccb82ff"}
Oct 30 18:52:43.560064 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.559564    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"
Oct 30 18:52:43.643705 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.643645    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs" podStartSLOduration=3.177424042 podStartE2EDuration="5.643625735s" totalImagesPullingTime="2.466201693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:43.636865331 +0000 UTC m=+1177.505283662" watchObservedRunningTime="2026-10-30 18:52:43.643625735 +0000 UTC m=+1177.512044067"
Oct 30 18:52:43.708203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.701096    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"
Oct 30 18:52:43.960427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.957049    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:52:43.960427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.957182    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 18:52:43.960427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:43.957229    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc" podUID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 18:52:44.216639 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:44.216551    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"]
Oct 30 18:52:44.295472 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:44.294581    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6b55fc-3dab-4085-9f1c-672721d0538f" path="/var/lib/kubelet/pods/ea6b55fc-3dab-4085-9f1c-672721d0538f/volumes"
Oct 30 18:52:44.575263 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:44.575224    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" event={"ID":"2313cd49-c455-47e7-94df-091ce8c2edf9","Type":"ContainerStarted","Data":"0f03dc031131119f1405aefe91eef652f8f19235aca355a55bdde1bc94cc8f9a"}
Oct 30 18:52:44.576985 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:44.576959    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr" event={"ID":"6f837717-c395-406e-af47-b93f2213a2bb","Type":"ContainerStarted","Data":"82421493fd9d2963900a48d03ad83b5a5e09a2218e0337bcf837b0528f0f898e"}
Oct 30 18:52:44.577116 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:44.577006    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr" event={"ID":"6f837717-c395-406e-af47-b93f2213a2bb","Type":"ContainerStarted","Data":"2b0db1078b283ca5940b3f00ae58de249dfc12103cbb827a28702c8ca35f12bb"}
Oct 30 18:52:44.577116 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:44.577066    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"
Oct 30 18:52:44.656265 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:44.656204    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr" podStartSLOduration=1.6561856430000002 podStartE2EDuration="1.656185643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:44.652922383 +0000 UTC m=+1178.521340716" watchObservedRunningTime="2026-10-30 18:52:44.656185643 +0000 UTC m=+1178.524603974"
Oct 30 18:52:44.950947 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:44.950855    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 18:52:44.951223 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:44.951196    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc" podUID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 18:52:45.443514 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:45.443430    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:52:45.443514 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:45.443484    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 18:52:45.443736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:45.443556    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 18:52:45.950684 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:45.950647    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 18:52:45.951239 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:45.950723    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc" podUID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 18:52:46.447354 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:46.447317    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 18:52:46.447653 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:46.447626    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 18:52:46.976615 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:46.976577    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 18:52:47.004237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:47.004193    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc" podUID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 18:52:47.443046 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:47.442997    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 18:52:47.443251 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:47.443056    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 18:52:47.950621 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:47.950583    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 18:52:47.950828 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:47.950636    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc" podUID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 18:52:48.446648 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:48.446604    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 18:52:48.447615 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:48.447582    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 18:52:48.950635 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:48.950599    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 18:52:48.950824 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:48.950654    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc" podUID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 18:52:49.444081 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:49.444043    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 18:52:49.444330 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:49.444142    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 18:52:49.955159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:49.955106    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:52:49.955682 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:49.955664    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:52:49.956455 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:49.956434    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:52:50.443281 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:50.443244    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 18:52:50.443474 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:50.443310    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 18:52:51.447213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:51.447183    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:52:51.447679 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:51.447657    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:52:51.448304 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:51.448286    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:52:54.579567 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:54.579539    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"
Oct 30 18:52:55.658019 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:52:55.657986    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"
Oct 30 18:52:57.695907 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:52:57.695874    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:53:01.957160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:01.956918    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-998xh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:53:01.957160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:01.956987    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-998xh" podUID="ccc2f38a-84e5-43c3-98d3-0f39ea658a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:02.165296 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:02.165255    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qstgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:53:02.165482 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:02.165313    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qstgx" podUID="ce1f4775-99b1-4896-b58f-348c46e3b5f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:03.696168 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.696116    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"]
Oct 30 18:53:03.696937 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.696916    4372 cpu_manager.go:414] "containerMap: removing container" podUID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerName="istio-proxy"
Oct 30 18:53:03.697049 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.696940    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerName="istio-proxy"
Oct 30 18:53:03.697183 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.697168    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf281ca-692b-48c8-84d7-13ca8e6e7489" containerName="istio-proxy"
Oct 30 18:53:03.700919 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.700899    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.715339 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.715319    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-348877-openshift-default-dockercfg-lk9hh\""
Oct 30 18:53:03.725719 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.725692    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"]
Oct 30 18:53:03.767146 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.767099    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-credential-socket\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.767383 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.767355    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-envoy\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.767550 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.767531    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-data\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.767643 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.767627    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-podinfo\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.767715 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.767700    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istiod-ca-cert\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.767816 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.767802    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-workload-certs\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.767871 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.767850    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-workload-socket\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.767920 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.767882    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-token\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.768027 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.767966    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlgfl\" (UniqueName: \"kubernetes.io/projected/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-kube-api-access-nlgfl\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.962205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.947516    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-workload-certs\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.962205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.947588    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-workload-socket\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.962205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.947624    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-token\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.962205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.947708    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlgfl\" (UniqueName: \"kubernetes.io/projected/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-kube-api-access-nlgfl\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.962205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.947790    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-credential-socket\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.962205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.947833    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-envoy\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.962205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.947968    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-data\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.962205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.948083    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-podinfo\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.962205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.948138    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istiod-ca-cert\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.962205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.948992    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istiod-ca-cert\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.962205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.949301    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-workload-certs\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.962205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.949523    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-workload-socket\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:03.967623 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:03.967593    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-data\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:04.042591 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:04.042559    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlgfl\" (UniqueName: \"kubernetes.io/projected/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-kube-api-access-nlgfl\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:04.049214 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:04.043642    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-token\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:04.070043 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:04.045029    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-envoy\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:04.070274 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:04.057644    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-credential-socket\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:04.082262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:04.080834    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-podinfo\") pod \"e2e-ipp-b-348877-openshift-default-5567478d87-mkj74\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") " pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:04.314959 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:04.314875    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:04.603347 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:04.603323    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"]
Oct 30 18:53:04.603923 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:53:04.603901    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b527cf_ac31_44b7_8ff0_6e40ee60ce32.slice/crio-a1121c73adfa343b1d5d69b1cae4de4fbe806ed0311768907bda0125872e1497 WatchSource:0}: Error finding container a1121c73adfa343b1d5d69b1cae4de4fbe806ed0311768907bda0125872e1497: Status 404 returned error can't find the container with id a1121c73adfa343b1d5d69b1cae4de4fbe806ed0311768907bda0125872e1497
Oct 30 18:53:04.606476 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:04.606445    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:04.606562 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:04.606513    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:04.606562 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:04.606543    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:05.046201 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:05.046080    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74" event={"ID":"57b527cf-ac31-44b7-8ff0-6e40ee60ce32","Type":"ContainerStarted","Data":"ad245b65e6a015a18a9ba10eafe03212a0977da46bee1264b343f742b4563490"}
Oct 30 18:53:05.046201 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:05.046157    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74" event={"ID":"57b527cf-ac31-44b7-8ff0-6e40ee60ce32","Type":"ContainerStarted","Data":"a1121c73adfa343b1d5d69b1cae4de4fbe806ed0311768907bda0125872e1497"}
Oct 30 18:53:05.330515 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:05.329232    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:05.357877 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:05.354668    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:05.381634 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:05.381585    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74" podStartSLOduration=2.381570442 podStartE2EDuration="2.381570442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:05.150827314 +0000 UTC m=+1199.019245645" watchObservedRunningTime="2026-10-30 18:53:05.381570442 +0000 UTC m=+1199.249988762"
Oct 30 18:53:06.057024 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:06.056994    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:06.058030 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:06.058009    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:53:08.845472 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:08.845438    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"]
Oct 30 18:53:08.851429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:08.851404    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"
Oct 30 18:53:08.867465 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:08.867441    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-348877\""
Oct 30 18:53:08.873075 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:08.873048    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"]
Oct 30 18:53:08.935907 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:08.935880    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-348877-dockercfg-grl5w\""
Oct 30 18:53:08.956804 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:08.956776    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49cqq\" (UniqueName: \"kubernetes.io/projected/5ea0460f-ea81-4c9a-af83-cfb66861b890-kube-api-access-49cqq\") pod \"payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw\" (UID: \"5ea0460f-ea81-4c9a-af83-cfb66861b890\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"
Oct 30 18:53:08.957158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:08.957109    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea0460f-ea81-4c9a-af83-cfb66861b890-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw\" (UID: \"5ea0460f-ea81-4c9a-af83-cfb66861b890\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"
Oct 30 18:53:09.068441 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:09.068401    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea0460f-ea81-4c9a-af83-cfb66861b890-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw\" (UID: \"5ea0460f-ea81-4c9a-af83-cfb66861b890\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"
Oct 30 18:53:09.069728 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:09.069704    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49cqq\" (UniqueName: \"kubernetes.io/projected/5ea0460f-ea81-4c9a-af83-cfb66861b890-kube-api-access-49cqq\") pod \"payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw\" (UID: \"5ea0460f-ea81-4c9a-af83-cfb66861b890\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"
Oct 30 18:53:09.070369 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:09.069493    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea0460f-ea81-4c9a-af83-cfb66861b890-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw\" (UID: \"5ea0460f-ea81-4c9a-af83-cfb66861b890\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"
Oct 30 18:53:09.122293 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:09.122221    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49cqq\" (UniqueName: \"kubernetes.io/projected/5ea0460f-ea81-4c9a-af83-cfb66861b890-kube-api-access-49cqq\") pod \"payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw\" (UID: \"5ea0460f-ea81-4c9a-af83-cfb66861b890\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"
Oct 30 18:53:09.167352 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:09.167310    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"
Oct 30 18:53:09.565211 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:53:09.565177    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea0460f_ea81_4c9a_af83_cfb66861b890.slice/crio-f677c0d38ade07581e29b6542cb614082c20c78a08db475e6a24772b15fbc6ad WatchSource:0}: Error finding container f677c0d38ade07581e29b6542cb614082c20c78a08db475e6a24772b15fbc6ad: Status 404 returned error can't find the container with id f677c0d38ade07581e29b6542cb614082c20c78a08db475e6a24772b15fbc6ad
Oct 30 18:53:09.565509 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:09.565492    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"]
Oct 30 18:53:10.134036 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.134001    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw" event={"ID":"5ea0460f-ea81-4c9a-af83-cfb66861b890","Type":"ContainerStarted","Data":"552e6dffc6c54d0e2c7cef9111228383e1eaecfa636f828fa6b2189d893d88c2"}
Oct 30 18:53:10.134477 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.134060    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw" event={"ID":"5ea0460f-ea81-4c9a-af83-cfb66861b890","Type":"ContainerStarted","Data":"f677c0d38ade07581e29b6542cb614082c20c78a08db475e6a24772b15fbc6ad"}
Oct 30 18:53:10.134477 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.134231    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"
Oct 30 18:53:10.158593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.158551    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw" podStartSLOduration=2.158540454 podStartE2EDuration="2.158540454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:10.155654875 +0000 UTC m=+1204.024073207" watchObservedRunningTime="2026-10-30 18:53:10.158540454 +0000 UTC m=+1204.026958788"
Oct 30 18:53:10.277994 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.277960    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"]
Oct 30 18:53:10.278242 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.278208    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j" podUID="239a7d10-93ea-4552-bd23-50bcda6292e5" containerName="maas-api" containerID="cri-o://c25d4676e6d07348c0e35fd7c0536026a970440f7b6831c4405ae80c081ebf46" gracePeriod=30
Oct 30 18:53:10.597258 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.597231    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:53:10.702480 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.702405    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239a7d10-93ea-4552-bd23-50bcda6292e5-kube-api-access-9ptrg\" (UniqueName: \"kubernetes.io/projected/239a7d10-93ea-4552-bd23-50bcda6292e5-kube-api-access-9ptrg\") pod \"239a7d10-93ea-4552-bd23-50bcda6292e5\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") "
Oct 30 18:53:10.702638 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.702502    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/239a7d10-93ea-4552-bd23-50bcda6292e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/239a7d10-93ea-4552-bd23-50bcda6292e5-maas-api-tls\") pod \"239a7d10-93ea-4552-bd23-50bcda6292e5\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") "
Oct 30 18:53:10.702700 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.702663    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/239a7d10-93ea-4552-bd23-50bcda6292e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/239a7d10-93ea-4552-bd23-50bcda6292e5-metrics-certs\") pod \"239a7d10-93ea-4552-bd23-50bcda6292e5\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") "
Oct 30 18:53:10.702800 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.702766    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/239a7d10-93ea-4552-bd23-50bcda6292e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/239a7d10-93ea-4552-bd23-50bcda6292e5-openshift-service-ca\") pod \"239a7d10-93ea-4552-bd23-50bcda6292e5\" (UID: \"239a7d10-93ea-4552-bd23-50bcda6292e5\") "
Oct 30 18:53:10.703580 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.703158    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/239a7d10-93ea-4552-bd23-50bcda6292e5-openshift-service-ca" pod "239a7d10-93ea-4552-bd23-50bcda6292e5" (UID: "239a7d10-93ea-4552-bd23-50bcda6292e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:10.703580 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.703574    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/239a7d10-93ea-4552-bd23-50bcda6292e5-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:10.704847 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.704827    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/239a7d10-93ea-4552-bd23-50bcda6292e5-maas-api-tls" pod "239a7d10-93ea-4552-bd23-50bcda6292e5" (UID: "239a7d10-93ea-4552-bd23-50bcda6292e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:10.705570 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.705547    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/239a7d10-93ea-4552-bd23-50bcda6292e5-metrics-certs" pod "239a7d10-93ea-4552-bd23-50bcda6292e5" (UID: "239a7d10-93ea-4552-bd23-50bcda6292e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:10.705650 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.705625    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239a7d10-93ea-4552-bd23-50bcda6292e5-kube-api-access-9ptrg" pod "239a7d10-93ea-4552-bd23-50bcda6292e5" (UID: "239a7d10-93ea-4552-bd23-50bcda6292e5"). InnerVolumeSpecName "kube-api-access-9ptrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:10.810191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.805408    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/239a7d10-93ea-4552-bd23-50bcda6292e5-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:10.810191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.805447    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ptrg\" (UniqueName: \"kubernetes.io/projected/239a7d10-93ea-4552-bd23-50bcda6292e5-kube-api-access-9ptrg\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:10.810191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:10.805462    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/239a7d10-93ea-4552-bd23-50bcda6292e5-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:11.227150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:11.219258    4372 generic.go:356] "Generic (PLEG): container finished" podID="239a7d10-93ea-4552-bd23-50bcda6292e5" containerID="c25d4676e6d07348c0e35fd7c0536026a970440f7b6831c4405ae80c081ebf46" exitCode=0
Oct 30 18:53:11.227150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:11.220515    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"
Oct 30 18:53:11.227150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:11.225927    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j" event={"ID":"239a7d10-93ea-4552-bd23-50bcda6292e5","Type":"ContainerDied","Data":"c25d4676e6d07348c0e35fd7c0536026a970440f7b6831c4405ae80c081ebf46"}
Oct 30 18:53:11.227150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:11.225985    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j" event={"ID":"239a7d10-93ea-4552-bd23-50bcda6292e5","Type":"ContainerDied","Data":"3a9af40e8017bd951111f1b4d6068f9af3e42178ef80c85c3d8f925a02b8d5c2"}
Oct 30 18:53:11.227150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:11.226014    4372 scope.go:122] "RemoveContainer" containerID="c25d4676e6d07348c0e35fd7c0536026a970440f7b6831c4405ae80c081ebf46"
Oct 30 18:53:11.261399 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:11.257116    4372 scope.go:122] "RemoveContainer" containerID="c25d4676e6d07348c0e35fd7c0536026a970440f7b6831c4405ae80c081ebf46"
Oct 30 18:53:11.261399 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:53:11.259221    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c25d4676e6d07348c0e35fd7c0536026a970440f7b6831c4405ae80c081ebf46\": container with ID starting with c25d4676e6d07348c0e35fd7c0536026a970440f7b6831c4405ae80c081ebf46 not found: ID does not exist" containerID="c25d4676e6d07348c0e35fd7c0536026a970440f7b6831c4405ae80c081ebf46"
Oct 30 18:53:11.261399 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:11.259257    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25d4676e6d07348c0e35fd7c0536026a970440f7b6831c4405ae80c081ebf46"} err="failed to get container status \"c25d4676e6d07348c0e35fd7c0536026a970440f7b6831c4405ae80c081ebf46\": rpc error: code = NotFound desc = could not find container \"c25d4676e6d07348c0e35fd7c0536026a970440f7b6831c4405ae80c081ebf46\": container with ID starting with c25d4676e6d07348c0e35fd7c0536026a970440f7b6831c4405ae80c081ebf46 not found: ID does not exist"
Oct 30 18:53:11.284917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:11.284880    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"]
Oct 30 18:53:11.291240 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:11.291206    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-52fa8b-55cfdfc45f-5sh8j"]
Oct 30 18:53:11.877525 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:11.877498    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"]
Oct 30 18:53:11.877731 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:11.877703    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8" podUID="d1fbc536-f3af-43db-8bf9-067a4c6598c8" containerName="maas-api" containerID="cri-o://8c028cd05685b29b46e44f7120f5b8df49ae90e006167d2bd5952f4ce65b3d14" gracePeriod=30
Oct 30 18:53:12.060449 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.060412    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"]
Oct 30 18:53:12.060710 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.060694    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0362f6d7-55jjw"]
Oct 30 18:53:12.194260 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.194229    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:53:12.321719 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.321684    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10da9fc4-c422-4802-96e4-222a3274b87d" path="/var/lib/kubelet/pods/10da9fc4-c422-4802-96e4-222a3274b87d/volumes"
Oct 30 18:53:12.323027 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.323001    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="239a7d10-93ea-4552-bd23-50bcda6292e5" path="/var/lib/kubelet/pods/239a7d10-93ea-4552-bd23-50bcda6292e5/volumes"
Oct 30 18:53:12.363149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.362923    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1fbc536-f3af-43db-8bf9-067a4c6598c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1fbc536-f3af-43db-8bf9-067a4c6598c8-metrics-certs\") pod \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") "
Oct 30 18:53:12.363149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.362981    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1fbc536-f3af-43db-8bf9-067a4c6598c8-kube-api-access-hh6tf\" (UniqueName: \"kubernetes.io/projected/d1fbc536-f3af-43db-8bf9-067a4c6598c8-kube-api-access-hh6tf\") pod \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") "
Oct 30 18:53:12.363149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.363032    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1fbc536-f3af-43db-8bf9-067a4c6598c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1fbc536-f3af-43db-8bf9-067a4c6598c8-maas-api-tls\") pod \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") "
Oct 30 18:53:12.363149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.363066    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1fbc536-f3af-43db-8bf9-067a4c6598c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1fbc536-f3af-43db-8bf9-067a4c6598c8-openshift-service-ca\") pod \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\" (UID: \"d1fbc536-f3af-43db-8bf9-067a4c6598c8\") "
Oct 30 18:53:12.461669 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.448491    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1fbc536-f3af-43db-8bf9-067a4c6598c8-openshift-service-ca" pod "d1fbc536-f3af-43db-8bf9-067a4c6598c8" (UID: "d1fbc536-f3af-43db-8bf9-067a4c6598c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:12.521151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.521082    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fbc536-f3af-43db-8bf9-067a4c6598c8-maas-api-tls" pod "d1fbc536-f3af-43db-8bf9-067a4c6598c8" (UID: "d1fbc536-f3af-43db-8bf9-067a4c6598c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:12.540875 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.533873    4372 generic.go:356] "Generic (PLEG): container finished" podID="d1fbc536-f3af-43db-8bf9-067a4c6598c8" containerID="8c028cd05685b29b46e44f7120f5b8df49ae90e006167d2bd5952f4ce65b3d14" exitCode=0
Oct 30 18:53:12.540875 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.533989    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8" event={"ID":"d1fbc536-f3af-43db-8bf9-067a4c6598c8","Type":"ContainerDied","Data":"8c028cd05685b29b46e44f7120f5b8df49ae90e006167d2bd5952f4ce65b3d14"}
Oct 30 18:53:12.540875 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.534030    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8" event={"ID":"d1fbc536-f3af-43db-8bf9-067a4c6598c8","Type":"ContainerDied","Data":"642b96dc5b4b58e149caee934b0f11e2189768800d9189623cc920e46e4dab59"}
Oct 30 18:53:12.540875 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.534055    4372 scope.go:122] "RemoveContainer" containerID="8c028cd05685b29b46e44f7120f5b8df49ae90e006167d2bd5952f4ce65b3d14"
Oct 30 18:53:12.540875 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.534201    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"
Oct 30 18:53:12.551405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.551373    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fbc536-f3af-43db-8bf9-067a4c6598c8-metrics-certs" pod "d1fbc536-f3af-43db-8bf9-067a4c6598c8" (UID: "d1fbc536-f3af-43db-8bf9-067a4c6598c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:12.563180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.552807    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1fbc536-f3af-43db-8bf9-067a4c6598c8-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:12.567003 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.566980    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1fbc536-f3af-43db-8bf9-067a4c6598c8-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:12.567199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.567184    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1fbc536-f3af-43db-8bf9-067a4c6598c8-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:12.568339 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.568312    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fbc536-f3af-43db-8bf9-067a4c6598c8-kube-api-access-hh6tf" pod "d1fbc536-f3af-43db-8bf9-067a4c6598c8" (UID: "d1fbc536-f3af-43db-8bf9-067a4c6598c8"). InnerVolumeSpecName "kube-api-access-hh6tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:12.610351 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.602695    4372 scope.go:122] "RemoveContainer" containerID="8c028cd05685b29b46e44f7120f5b8df49ae90e006167d2bd5952f4ce65b3d14"
Oct 30 18:53:12.610351 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:53:12.603247    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c028cd05685b29b46e44f7120f5b8df49ae90e006167d2bd5952f4ce65b3d14\": container with ID starting with 8c028cd05685b29b46e44f7120f5b8df49ae90e006167d2bd5952f4ce65b3d14 not found: ID does not exist" containerID="8c028cd05685b29b46e44f7120f5b8df49ae90e006167d2bd5952f4ce65b3d14"
Oct 30 18:53:12.610351 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.603283    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c028cd05685b29b46e44f7120f5b8df49ae90e006167d2bd5952f4ce65b3d14"} err="failed to get container status \"8c028cd05685b29b46e44f7120f5b8df49ae90e006167d2bd5952f4ce65b3d14\": rpc error: code = NotFound desc = could not find container \"8c028cd05685b29b46e44f7120f5b8df49ae90e006167d2bd5952f4ce65b3d14\": container with ID starting with 8c028cd05685b29b46e44f7120f5b8df49ae90e006167d2bd5952f4ce65b3d14 not found: ID does not exist"
Oct 30 18:53:12.672219 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.669337    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh6tf\" (UniqueName: \"kubernetes.io/projected/d1fbc536-f3af-43db-8bf9-067a4c6598c8-kube-api-access-hh6tf\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:12.866768 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.866737    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"]
Oct 30 18:53:12.877398 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:12.877376    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0d540b-7c9bf7cbdd-dcwc8"]
Oct 30 18:53:14.266611 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:14.266579    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1fbc536-f3af-43db-8bf9-067a4c6598c8" path="/var/lib/kubelet/pods/d1fbc536-f3af-43db-8bf9-067a4c6598c8/volumes"
Oct 30 18:53:18.944428 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:18.944341    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"]
Oct 30 18:53:18.944920 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:18.944539    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t" podUID="0328c404-96a0-447a-b06c-a1ef68ba3178" containerName="istio-proxy" containerID="cri-o://f3cc23758dabf5a8524a34d75ff60165d1154fa1296896275ce58f83d820765b" gracePeriod=30
Oct 30 18:53:21.223471 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:21.223436    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"
Oct 30 18:53:24.573894 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.570463    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:53:24.818810 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.818781    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:53:24.878714 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.878486    4372 generic.go:356] "Generic (PLEG): container finished" podID="0328c404-96a0-447a-b06c-a1ef68ba3178" containerID="f3cc23758dabf5a8524a34d75ff60165d1154fa1296896275ce58f83d820765b" exitCode=0
Oct 30 18:53:24.878714 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.878550    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"
Oct 30 18:53:24.878974 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.878537    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t" event={"ID":"0328c404-96a0-447a-b06c-a1ef68ba3178","Type":"ContainerDied","Data":"f3cc23758dabf5a8524a34d75ff60165d1154fa1296896275ce58f83d820765b"}
Oct 30 18:53:24.878974 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.878785    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t" event={"ID":"0328c404-96a0-447a-b06c-a1ef68ba3178","Type":"ContainerDied","Data":"944208a83081e87886991211c4deb47858abad820646f05e76a2750cad02d50a"}
Oct 30 18:53:24.878974 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.878816    4372 scope.go:122] "RemoveContainer" containerID="f3cc23758dabf5a8524a34d75ff60165d1154fa1296896275ce58f83d820765b"
Oct 30 18:53:24.923239 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.923203    4372 scope.go:122] "RemoveContainer" containerID="cdf726eec866bac5ec4c9315877bb58e09621cda2eca0f6c018ed444b0910714"
Oct 30 18:53:24.935279 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.935259    4372 scope.go:122] "RemoveContainer" containerID="f3cc23758dabf5a8524a34d75ff60165d1154fa1296896275ce58f83d820765b"
Oct 30 18:53:24.935595 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:53:24.935575    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3cc23758dabf5a8524a34d75ff60165d1154fa1296896275ce58f83d820765b\": container with ID starting with f3cc23758dabf5a8524a34d75ff60165d1154fa1296896275ce58f83d820765b not found: ID does not exist" containerID="f3cc23758dabf5a8524a34d75ff60165d1154fa1296896275ce58f83d820765b"
Oct 30 18:53:24.935706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.935602    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3cc23758dabf5a8524a34d75ff60165d1154fa1296896275ce58f83d820765b"} err="failed to get container status \"f3cc23758dabf5a8524a34d75ff60165d1154fa1296896275ce58f83d820765b\": rpc error: code = NotFound desc = could not find container \"f3cc23758dabf5a8524a34d75ff60165d1154fa1296896275ce58f83d820765b\": container with ID starting with f3cc23758dabf5a8524a34d75ff60165d1154fa1296896275ce58f83d820765b not found: ID does not exist"
Oct 30 18:53:24.935706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.935620    4372 scope.go:122] "RemoveContainer" containerID="cdf726eec866bac5ec4c9315877bb58e09621cda2eca0f6c018ed444b0910714"
Oct 30 18:53:24.935904 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:53:24.935890    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf726eec866bac5ec4c9315877bb58e09621cda2eca0f6c018ed444b0910714\": container with ID starting with cdf726eec866bac5ec4c9315877bb58e09621cda2eca0f6c018ed444b0910714 not found: ID does not exist" containerID="cdf726eec866bac5ec4c9315877bb58e09621cda2eca0f6c018ed444b0910714"
Oct 30 18:53:24.935950 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.935909    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf726eec866bac5ec4c9315877bb58e09621cda2eca0f6c018ed444b0910714"} err="failed to get container status \"cdf726eec866bac5ec4c9315877bb58e09621cda2eca0f6c018ed444b0910714\": rpc error: code = NotFound desc = could not find container \"cdf726eec866bac5ec4c9315877bb58e09621cda2eca0f6c018ed444b0910714\": container with ID starting with cdf726eec866bac5ec4c9315877bb58e09621cda2eca0f6c018ed444b0910714 not found: ID does not exist"
Oct 30 18:53:24.950175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.950121    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-credential-socket\") pod \"0328c404-96a0-447a-b06c-a1ef68ba3178\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") "
Oct 30 18:53:24.950311 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.950219    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-podinfo\") pod \"0328c404-96a0-447a-b06c-a1ef68ba3178\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") "
Oct 30 18:53:24.950311 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.950246    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0328c404-96a0-447a-b06c-a1ef68ba3178-kube-api-access-fxrdx\" (UniqueName: \"kubernetes.io/projected/0328c404-96a0-447a-b06c-a1ef68ba3178-kube-api-access-fxrdx\") pod \"0328c404-96a0-447a-b06c-a1ef68ba3178\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") "
Oct 30 18:53:24.950311 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.950275    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0328c404-96a0-447a-b06c-a1ef68ba3178-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0328c404-96a0-447a-b06c-a1ef68ba3178-istiod-ca-cert\") pod \"0328c404-96a0-447a-b06c-a1ef68ba3178\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") "
Oct 30 18:53:24.950436 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.950366    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-envoy\") pod \"0328c404-96a0-447a-b06c-a1ef68ba3178\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") "
Oct 30 18:53:24.950436 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.950373    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-credential-socket" pod "0328c404-96a0-447a-b06c-a1ef68ba3178" (UID: "0328c404-96a0-447a-b06c-a1ef68ba3178"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:24.950436 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.950400    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-workload-certs\") pod \"0328c404-96a0-447a-b06c-a1ef68ba3178\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") "
Oct 30 18:53:24.950574 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.950557    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-data\") pod \"0328c404-96a0-447a-b06c-a1ef68ba3178\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") "
Oct 30 18:53:24.950628 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.950609    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-token\" (UniqueName: \"kubernetes.io/projected/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-token\") pod \"0328c404-96a0-447a-b06c-a1ef68ba3178\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") "
Oct 30 18:53:24.950729 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.950705    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0328c404-96a0-447a-b06c-a1ef68ba3178-istiod-ca-cert" pod "0328c404-96a0-447a-b06c-a1ef68ba3178" (UID: "0328c404-96a0-447a-b06c-a1ef68ba3178"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:24.950789 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.950709    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-workload-socket\") pod \"0328c404-96a0-447a-b06c-a1ef68ba3178\" (UID: \"0328c404-96a0-447a-b06c-a1ef68ba3178\") "
Oct 30 18:53:24.950969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.950947    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-workload-socket" pod "0328c404-96a0-447a-b06c-a1ef68ba3178" (UID: "0328c404-96a0-447a-b06c-a1ef68ba3178"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:24.951361 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.951342    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-workload-certs" pod "0328c404-96a0-447a-b06c-a1ef68ba3178" (UID: "0328c404-96a0-447a-b06c-a1ef68ba3178"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:24.951560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.951537    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-data" pod "0328c404-96a0-447a-b06c-a1ef68ba3178" (UID: "0328c404-96a0-447a-b06c-a1ef68ba3178"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:24.951699 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.951685    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:24.951770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.951704    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:24.951770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.951713    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:24.951770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.951723    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0328c404-96a0-447a-b06c-a1ef68ba3178-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:24.951770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.951731    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:24.953688 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.953664    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-token" pod "0328c404-96a0-447a-b06c-a1ef68ba3178" (UID: "0328c404-96a0-447a-b06c-a1ef68ba3178"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:24.954607 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.954585    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-envoy" pod "0328c404-96a0-447a-b06c-a1ef68ba3178" (UID: "0328c404-96a0-447a-b06c-a1ef68ba3178"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:24.955115 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.955084    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0328c404-96a0-447a-b06c-a1ef68ba3178-kube-api-access-fxrdx" pod "0328c404-96a0-447a-b06c-a1ef68ba3178" (UID: "0328c404-96a0-447a-b06c-a1ef68ba3178"). InnerVolumeSpecName "kube-api-access-fxrdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:24.955822 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:24.955802    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-podinfo" pod "0328c404-96a0-447a-b06c-a1ef68ba3178" (UID: "0328c404-96a0-447a-b06c-a1ef68ba3178"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:53:25.053173 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:25.053143    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:25.053173 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:25.053169    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxrdx\" (UniqueName: \"kubernetes.io/projected/0328c404-96a0-447a-b06c-a1ef68ba3178-kube-api-access-fxrdx\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:25.053363 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:25.053181    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:25.053363 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:25.053193    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0328c404-96a0-447a-b06c-a1ef68ba3178-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:25.056270 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:25.056248    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"]
Oct 30 18:53:25.056490 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:25.056466    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h" podUID="5a4347ef-492a-4569-a344-1e683419ea7f" containerName="istio-proxy" containerID="cri-o://0174c36ba46f468afea05521fa725c6c9ca6559ff1ed5ca70e908fd3da0d8712" gracePeriod=30
Oct 30 18:53:25.333152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:25.330191    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"]
Oct 30 18:53:25.333152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:25.331497    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-52fa8b-openshift-default-7ddc6d5575-kjd7t"]
Oct 30 18:53:26.268394 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:26.268364    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0328c404-96a0-447a-b06c-a1ef68ba3178" path="/var/lib/kubelet/pods/0328c404-96a0-447a-b06c-a1ef68ba3178/volumes"
Oct 30 18:53:27.695936 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:53:27.695902    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:53:29.344855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:29.344818    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:53:29.345369 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:29.344877    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h" podUID="5a4347ef-492a-4569-a344-1e683419ea7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:53:30.350920 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.350896    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:53:30.462530 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.462498    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a4347ef-492a-4569-a344-1e683419ea7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4347ef-492a-4569-a344-1e683419ea7f-istiod-ca-cert\") pod \"5a4347ef-492a-4569-a344-1e683419ea7f\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") "
Oct 30 18:53:30.462530 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.462534    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4347ef-492a-4569-a344-1e683419ea7f-kube-api-access-msmsz\" (UniqueName: \"kubernetes.io/projected/5a4347ef-492a-4569-a344-1e683419ea7f-kube-api-access-msmsz\") pod \"5a4347ef-492a-4569-a344-1e683419ea7f\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") "
Oct 30 18:53:30.462742 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.462565    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-credential-socket\") pod \"5a4347ef-492a-4569-a344-1e683419ea7f\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") "
Oct 30 18:53:30.462742 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.462713    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-workload-certs\") pod \"5a4347ef-492a-4569-a344-1e683419ea7f\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") "
Oct 30 18:53:30.462885 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.462761    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a4347ef-492a-4569-a344-1e683419ea7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4347ef-492a-4569-a344-1e683419ea7f-istio-podinfo\") pod \"5a4347ef-492a-4569-a344-1e683419ea7f\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") "
Oct 30 18:53:30.462885 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.462864    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-credential-socket" pod "5a4347ef-492a-4569-a344-1e683419ea7f" (UID: "5a4347ef-492a-4569-a344-1e683419ea7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:30.462991 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.462932    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-istio-envoy\") pod \"5a4347ef-492a-4569-a344-1e683419ea7f\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") "
Oct 30 18:53:30.462991 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.462976    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-workload-certs" pod "5a4347ef-492a-4569-a344-1e683419ea7f" (UID: "5a4347ef-492a-4569-a344-1e683419ea7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:30.463089 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.463009    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4347ef-492a-4569-a344-1e683419ea7f-istiod-ca-cert" pod "5a4347ef-492a-4569-a344-1e683419ea7f" (UID: "5a4347ef-492a-4569-a344-1e683419ea7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:30.463089 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.463051    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-istio-data\") pod \"5a4347ef-492a-4569-a344-1e683419ea7f\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") "
Oct 30 18:53:30.463218 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.463120    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-workload-socket\") pod \"5a4347ef-492a-4569-a344-1e683419ea7f\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") "
Oct 30 18:53:30.463218 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.463192    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4347ef-492a-4569-a344-1e683419ea7f-istio-token\" (UniqueName: \"kubernetes.io/projected/5a4347ef-492a-4569-a344-1e683419ea7f-istio-token\") pod \"5a4347ef-492a-4569-a344-1e683419ea7f\" (UID: \"5a4347ef-492a-4569-a344-1e683419ea7f\") "
Oct 30 18:53:30.463407 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.463381    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-istio-data" pod "5a4347ef-492a-4569-a344-1e683419ea7f" (UID: "5a4347ef-492a-4569-a344-1e683419ea7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:30.463407 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.463395    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-workload-socket" pod "5a4347ef-492a-4569-a344-1e683419ea7f" (UID: "5a4347ef-492a-4569-a344-1e683419ea7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:30.464108 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.464063    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4347ef-492a-4569-a344-1e683419ea7f-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:30.464108 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.464096    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:30.464300 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.464116    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:30.464300 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.464149    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:30.464300 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.464164    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:30.465041 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.465010    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4347ef-492a-4569-a344-1e683419ea7f-kube-api-access-msmsz" pod "5a4347ef-492a-4569-a344-1e683419ea7f" (UID: "5a4347ef-492a-4569-a344-1e683419ea7f"). InnerVolumeSpecName "kube-api-access-msmsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:30.465194 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.465168    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a4347ef-492a-4569-a344-1e683419ea7f-istio-podinfo" pod "5a4347ef-492a-4569-a344-1e683419ea7f" (UID: "5a4347ef-492a-4569-a344-1e683419ea7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:53:30.465403 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.465384    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-istio-envoy" pod "5a4347ef-492a-4569-a344-1e683419ea7f" (UID: "5a4347ef-492a-4569-a344-1e683419ea7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:30.465648 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.465624    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4347ef-492a-4569-a344-1e683419ea7f-istio-token" pod "5a4347ef-492a-4569-a344-1e683419ea7f" (UID: "5a4347ef-492a-4569-a344-1e683419ea7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:30.565115 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.565088    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4347ef-492a-4569-a344-1e683419ea7f-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:30.565115 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.565116    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msmsz\" (UniqueName: \"kubernetes.io/projected/5a4347ef-492a-4569-a344-1e683419ea7f-kube-api-access-msmsz\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:30.565269 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.565149    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4347ef-492a-4569-a344-1e683419ea7f-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:30.565269 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.565161    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4347ef-492a-4569-a344-1e683419ea7f-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:30.942483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.942454    4372 generic.go:356] "Generic (PLEG): container finished" podID="5a4347ef-492a-4569-a344-1e683419ea7f" containerID="0174c36ba46f468afea05521fa725c6c9ca6559ff1ed5ca70e908fd3da0d8712" exitCode=0
Oct 30 18:53:30.942661 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.942523    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h" event={"ID":"5a4347ef-492a-4569-a344-1e683419ea7f","Type":"ContainerDied","Data":"0174c36ba46f468afea05521fa725c6c9ca6559ff1ed5ca70e908fd3da0d8712"}
Oct 30 18:53:30.942661 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.942575    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"
Oct 30 18:53:30.942661 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.942589    4372 scope.go:122] "RemoveContainer" containerID="0174c36ba46f468afea05521fa725c6c9ca6559ff1ed5ca70e908fd3da0d8712"
Oct 30 18:53:30.942797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.942579    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h" event={"ID":"5a4347ef-492a-4569-a344-1e683419ea7f","Type":"ContainerDied","Data":"59218e52c6ab0fdf7e5b1808f2f997c35bcf9adddffb7c6b12a64a191e753b93"}
Oct 30 18:53:30.954279 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.954264    4372 scope.go:122] "RemoveContainer" containerID="d5a3a7c509e4450c64837712c5c3c33c35c3f210794aea437c3b46f44e2e9137"
Oct 30 18:53:30.967457 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.966880    4372 scope.go:122] "RemoveContainer" containerID="0174c36ba46f468afea05521fa725c6c9ca6559ff1ed5ca70e908fd3da0d8712"
Oct 30 18:53:30.967457 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:53:30.967236    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0174c36ba46f468afea05521fa725c6c9ca6559ff1ed5ca70e908fd3da0d8712\": container with ID starting with 0174c36ba46f468afea05521fa725c6c9ca6559ff1ed5ca70e908fd3da0d8712 not found: ID does not exist" containerID="0174c36ba46f468afea05521fa725c6c9ca6559ff1ed5ca70e908fd3da0d8712"
Oct 30 18:53:30.967457 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.967268    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0174c36ba46f468afea05521fa725c6c9ca6559ff1ed5ca70e908fd3da0d8712"} err="failed to get container status \"0174c36ba46f468afea05521fa725c6c9ca6559ff1ed5ca70e908fd3da0d8712\": rpc error: code = NotFound desc = could not find container \"0174c36ba46f468afea05521fa725c6c9ca6559ff1ed5ca70e908fd3da0d8712\": container with ID starting with 0174c36ba46f468afea05521fa725c6c9ca6559ff1ed5ca70e908fd3da0d8712 not found: ID does not exist"
Oct 30 18:53:30.967457 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.967290    4372 scope.go:122] "RemoveContainer" containerID="d5a3a7c509e4450c64837712c5c3c33c35c3f210794aea437c3b46f44e2e9137"
Oct 30 18:53:30.967855 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:53:30.967833    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a3a7c509e4450c64837712c5c3c33c35c3f210794aea437c3b46f44e2e9137\": container with ID starting with d5a3a7c509e4450c64837712c5c3c33c35c3f210794aea437c3b46f44e2e9137 not found: ID does not exist" containerID="d5a3a7c509e4450c64837712c5c3c33c35c3f210794aea437c3b46f44e2e9137"
Oct 30 18:53:30.967909 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.967863    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a3a7c509e4450c64837712c5c3c33c35c3f210794aea437c3b46f44e2e9137"} err="failed to get container status \"d5a3a7c509e4450c64837712c5c3c33c35c3f210794aea437c3b46f44e2e9137\": rpc error: code = NotFound desc = could not find container \"d5a3a7c509e4450c64837712c5c3c33c35c3f210794aea437c3b46f44e2e9137\": container with ID starting with d5a3a7c509e4450c64837712c5c3c33c35c3f210794aea437c3b46f44e2e9137 not found: ID does not exist"
Oct 30 18:53:30.975622 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.975595    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"]
Oct 30 18:53:30.979116 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:30.979093    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5ad713-openshift-default-6db8ffb577-2gm7h"]
Oct 30 18:53:32.265509 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:32.265478    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4347ef-492a-4569-a344-1e683419ea7f" path="/var/lib/kubelet/pods/5a4347ef-492a-4569-a344-1e683419ea7f/volumes"
Oct 30 18:53:32.760155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:32.760011    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"]
Oct 30 18:53:34.445317 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:34.445286    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"]
Oct 30 18:53:34.445792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:34.445537    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr" podUID="6f837717-c395-406e-af47-b93f2213a2bb" containerName="payload-pre-processing" containerID="cri-o://82421493fd9d2963900a48d03ad83b5a5e09a2218e0337bcf837b0528f0f898e" gracePeriod=30
Oct 30 18:53:34.986302 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:34.986251    4372 generic.go:356] "Generic (PLEG): container finished" podID="6f837717-c395-406e-af47-b93f2213a2bb" containerID="82421493fd9d2963900a48d03ad83b5a5e09a2218e0337bcf837b0528f0f898e" exitCode=0
Oct 30 18:53:34.986452 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:34.986322    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr" event={"ID":"6f837717-c395-406e-af47-b93f2213a2bb","Type":"ContainerDied","Data":"82421493fd9d2963900a48d03ad83b5a5e09a2218e0337bcf837b0528f0f898e"}
Oct 30 18:53:35.020459 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:35.020437    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"
Oct 30 18:53:35.137002 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:35.136955    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f837717-c395-406e-af47-b93f2213a2bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f837717-c395-406e-af47-b93f2213a2bb-plugins-config-volume\") pod \"6f837717-c395-406e-af47-b93f2213a2bb\" (UID: \"6f837717-c395-406e-af47-b93f2213a2bb\") "
Oct 30 18:53:35.137215 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:35.137023    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f837717-c395-406e-af47-b93f2213a2bb-kube-api-access-75dn2\" (UniqueName: \"kubernetes.io/projected/6f837717-c395-406e-af47-b93f2213a2bb-kube-api-access-75dn2\") pod \"6f837717-c395-406e-af47-b93f2213a2bb\" (UID: \"6f837717-c395-406e-af47-b93f2213a2bb\") "
Oct 30 18:53:35.181018 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:35.180470    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f837717-c395-406e-af47-b93f2213a2bb-plugins-config-volume" pod "6f837717-c395-406e-af47-b93f2213a2bb" (UID: "6f837717-c395-406e-af47-b93f2213a2bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:35.187939 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:35.187903    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f837717-c395-406e-af47-b93f2213a2bb-kube-api-access-75dn2" pod "6f837717-c395-406e-af47-b93f2213a2bb" (UID: "6f837717-c395-406e-af47-b93f2213a2bb"). InnerVolumeSpecName "kube-api-access-75dn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:35.195286 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:35.194652    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f837717-c395-406e-af47-b93f2213a2bb-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:35.195286 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:35.194686    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75dn2\" (UniqueName: \"kubernetes.io/projected/6f837717-c395-406e-af47-b93f2213a2bb-kube-api-access-75dn2\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:35.993976 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:35.993940    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"
Oct 30 18:53:35.994437 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:35.993978    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr" event={"ID":"6f837717-c395-406e-af47-b93f2213a2bb","Type":"ContainerDied","Data":"2b0db1078b283ca5940b3f00ae58de249dfc12103cbb827a28702c8ca35f12bb"}
Oct 30 18:53:35.994437 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:35.994040    4372 scope.go:122] "RemoveContainer" containerID="82421493fd9d2963900a48d03ad83b5a5e09a2218e0337bcf837b0528f0f898e"
Oct 30 18:53:36.025567 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:36.025541    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"]
Oct 30 18:53:36.038640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:36.038616    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-594f74f4d7-zfhfr"]
Oct 30 18:53:36.266431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:36.266366    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f837717-c395-406e-af47-b93f2213a2bb" path="/var/lib/kubelet/pods/6f837717-c395-406e-af47-b93f2213a2bb/volumes"
Oct 30 18:53:37.761047 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:37.761006    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw" podUID="5ea0460f-ea81-4c9a-af83-cfb66861b890" containerName="payload-processing" containerID="cri-o://552e6dffc6c54d0e2c7cef9111228383e1eaecfa636f828fa6b2189d893d88c2" gracePeriod=25
Oct 30 18:53:38.021275 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:38.021196    4372 generic.go:356] "Generic (PLEG): container finished" podID="5ea0460f-ea81-4c9a-af83-cfb66861b890" containerID="552e6dffc6c54d0e2c7cef9111228383e1eaecfa636f828fa6b2189d893d88c2" exitCode=0
Oct 30 18:53:38.021451 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:38.021267    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw" event={"ID":"5ea0460f-ea81-4c9a-af83-cfb66861b890","Type":"ContainerDied","Data":"552e6dffc6c54d0e2c7cef9111228383e1eaecfa636f828fa6b2189d893d88c2"}
Oct 30 18:53:38.094682 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:38.094655    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"
Oct 30 18:53:38.268154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:38.255642    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea0460f-ea81-4c9a-af83-cfb66861b890-kube-api-access-49cqq\" (UniqueName: \"kubernetes.io/projected/5ea0460f-ea81-4c9a-af83-cfb66861b890-kube-api-access-49cqq\") pod \"5ea0460f-ea81-4c9a-af83-cfb66861b890\" (UID: \"5ea0460f-ea81-4c9a-af83-cfb66861b890\") "
Oct 30 18:53:38.268154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:38.255716    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea0460f-ea81-4c9a-af83-cfb66861b890-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea0460f-ea81-4c9a-af83-cfb66861b890-plugins-config-volume\") pod \"5ea0460f-ea81-4c9a-af83-cfb66861b890\" (UID: \"5ea0460f-ea81-4c9a-af83-cfb66861b890\") "
Oct 30 18:53:38.351113 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:38.345296    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea0460f-ea81-4c9a-af83-cfb66861b890-kube-api-access-49cqq" pod "5ea0460f-ea81-4c9a-af83-cfb66861b890" (UID: "5ea0460f-ea81-4c9a-af83-cfb66861b890"). InnerVolumeSpecName "kube-api-access-49cqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:38.351113 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:38.346285    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea0460f-ea81-4c9a-af83-cfb66861b890-plugins-config-volume" pod "5ea0460f-ea81-4c9a-af83-cfb66861b890" (UID: "5ea0460f-ea81-4c9a-af83-cfb66861b890"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:38.376152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:38.360497    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49cqq\" (UniqueName: \"kubernetes.io/projected/5ea0460f-ea81-4c9a-af83-cfb66861b890-kube-api-access-49cqq\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:38.389148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:38.376401    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea0460f-ea81-4c9a-af83-cfb66861b890-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:39.028910 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:39.028877    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"
Oct 30 18:53:39.029421 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:39.028910    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw" event={"ID":"5ea0460f-ea81-4c9a-af83-cfb66861b890","Type":"ContainerDied","Data":"f677c0d38ade07581e29b6542cb614082c20c78a08db475e6a24772b15fbc6ad"}
Oct 30 18:53:39.029421 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:39.028970    4372 scope.go:122] "RemoveContainer" containerID="552e6dffc6c54d0e2c7cef9111228383e1eaecfa636f828fa6b2189d893d88c2"
Oct 30 18:53:39.052599 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:39.052568    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"]
Oct 30 18:53:39.061010 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:39.060986    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-348877-846fd898d4-d99xw"]
Oct 30 18:53:40.265778 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:40.265745    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea0460f-ea81-4c9a-af83-cfb66861b890" path="/var/lib/kubelet/pods/5ea0460f-ea81-4c9a-af83-cfb66861b890/volumes"
Oct 30 18:53:41.235987 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.235956    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"]
Oct 30 18:53:41.236702 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236681    4372 cpu_manager.go:414] "containerMap: removing container" podUID="0328c404-96a0-447a-b06c-a1ef68ba3178" containerName="istio-proxy"
Oct 30 18:53:41.236702 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236701    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0328c404-96a0-447a-b06c-a1ef68ba3178" containerName="istio-proxy"
Oct 30 18:53:41.236917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236722    4372 cpu_manager.go:414] "containerMap: removing container" podUID="5a4347ef-492a-4569-a344-1e683419ea7f" containerName="istio-proxy"
Oct 30 18:53:41.236917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236728    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4347ef-492a-4569-a344-1e683419ea7f" containerName="istio-proxy"
Oct 30 18:53:41.236917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236741    4372 cpu_manager.go:414] "containerMap: removing container" podUID="239a7d10-93ea-4552-bd23-50bcda6292e5" containerName="maas-api"
Oct 30 18:53:41.236917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236747    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="239a7d10-93ea-4552-bd23-50bcda6292e5" containerName="maas-api"
Oct 30 18:53:41.236917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236755    4372 cpu_manager.go:414] "containerMap: removing container" podUID="5ea0460f-ea81-4c9a-af83-cfb66861b890" containerName="payload-processing"
Oct 30 18:53:41.236917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236760    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea0460f-ea81-4c9a-af83-cfb66861b890" containerName="payload-processing"
Oct 30 18:53:41.236917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236774    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d1fbc536-f3af-43db-8bf9-067a4c6598c8" containerName="maas-api"
Oct 30 18:53:41.236917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236779    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1fbc536-f3af-43db-8bf9-067a4c6598c8" containerName="maas-api"
Oct 30 18:53:41.236917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236789    4372 cpu_manager.go:414] "containerMap: removing container" podUID="6f837717-c395-406e-af47-b93f2213a2bb" containerName="payload-pre-processing"
Oct 30 18:53:41.236917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236797    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f837717-c395-406e-af47-b93f2213a2bb" containerName="payload-pre-processing"
Oct 30 18:53:41.236917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236894    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4347ef-492a-4569-a344-1e683419ea7f" containerName="istio-proxy"
Oct 30 18:53:41.236917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236906    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="239a7d10-93ea-4552-bd23-50bcda6292e5" containerName="maas-api"
Oct 30 18:53:41.236917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236914    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea0460f-ea81-4c9a-af83-cfb66861b890" containerName="payload-processing"
Oct 30 18:53:41.236917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236926    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1fbc536-f3af-43db-8bf9-067a4c6598c8" containerName="maas-api"
Oct 30 18:53:41.237696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236932    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f837717-c395-406e-af47-b93f2213a2bb" containerName="payload-pre-processing"
Oct 30 18:53:41.237696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236939    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="0328c404-96a0-447a-b06c-a1ef68ba3178" containerName="istio-proxy"
Oct 30 18:53:41.237696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.236951    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="0328c404-96a0-447a-b06c-a1ef68ba3178" containerName="istio-proxy"
Oct 30 18:53:41.242412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.242394    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"
Oct 30 18:53:41.248520 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.248498    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-348877\""
Oct 30 18:53:41.248668 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.248644    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-348877-dockercfg-n2nh4\""
Oct 30 18:53:41.253816 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.253794    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"]
Oct 30 18:53:41.427777 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.427178    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"]
Oct 30 18:53:41.462729 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.429141    4372 cpu_manager.go:414] "containerMap: removing container" podUID="0328c404-96a0-447a-b06c-a1ef68ba3178" containerName="istio-proxy"
Oct 30 18:53:41.462729 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.429169    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0328c404-96a0-447a-b06c-a1ef68ba3178" containerName="istio-proxy"
Oct 30 18:53:41.462729 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.429205    4372 cpu_manager.go:414] "containerMap: removing container" podUID="5a4347ef-492a-4569-a344-1e683419ea7f" containerName="istio-proxy"
Oct 30 18:53:41.462729 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.429213    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4347ef-492a-4569-a344-1e683419ea7f" containerName="istio-proxy"
Oct 30 18:53:41.462729 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.429390    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4347ef-492a-4569-a344-1e683419ea7f" containerName="istio-proxy"
Oct 30 18:53:41.500319 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.469005    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"
Oct 30 18:53:41.500319 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.496230    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"]
Oct 30 18:53:41.633219 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.630044    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"]
Oct 30 18:53:41.633219 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.630325    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh" podUID="bf1c2575-55bb-418a-8600-46e682bed18a" containerName="maas-api" containerID="cri-o://b848eea24c66522e873aac3b2b6477d7f52e589c0074d0a6fc8ab77ef14bf3e4" gracePeriod=30
Oct 30 18:53:41.633219 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.632601    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t79qr\" (UniqueName: \"kubernetes.io/projected/4fa388c0-7e72-4b86-b692-53d9c53ac30b-kube-api-access-t79qr\") pod \"payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg\" (UID: \"4fa388c0-7e72-4b86-b692-53d9c53ac30b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"
Oct 30 18:53:41.633219 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.632730    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71072275-42b7-4d5e-9e0c-cac53bf01426-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b\" (UID: \"71072275-42b7-4d5e-9e0c-cac53bf01426\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"
Oct 30 18:53:41.633219 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.632790    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa388c0-7e72-4b86-b692-53d9c53ac30b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg\" (UID: \"4fa388c0-7e72-4b86-b692-53d9c53ac30b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"
Oct 30 18:53:41.633219 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.632933    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6m9w\" (UniqueName: \"kubernetes.io/projected/71072275-42b7-4d5e-9e0c-cac53bf01426-kube-api-access-m6m9w\") pod \"payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b\" (UID: \"71072275-42b7-4d5e-9e0c-cac53bf01426\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"
Oct 30 18:53:41.647702 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.647670    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"]
Oct 30 18:53:41.647903 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.647867    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs" podUID="d6009aa5-288e-4eba-903a-7c4ad6a93c44" containerName="payload-processing" containerID="cri-o://cda298eededcb070261bf1d5828ebe0a612a991a0725879a620da48d5ccb82ff" gracePeriod=30
Oct 30 18:53:41.734203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.734163    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6m9w\" (UniqueName: \"kubernetes.io/projected/71072275-42b7-4d5e-9e0c-cac53bf01426-kube-api-access-m6m9w\") pod \"payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b\" (UID: \"71072275-42b7-4d5e-9e0c-cac53bf01426\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"
Oct 30 18:53:41.734369 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.734264    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t79qr\" (UniqueName: \"kubernetes.io/projected/4fa388c0-7e72-4b86-b692-53d9c53ac30b-kube-api-access-t79qr\") pod \"payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg\" (UID: \"4fa388c0-7e72-4b86-b692-53d9c53ac30b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"
Oct 30 18:53:41.734434 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.734365    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71072275-42b7-4d5e-9e0c-cac53bf01426-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b\" (UID: \"71072275-42b7-4d5e-9e0c-cac53bf01426\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"
Oct 30 18:53:41.734491 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.734430    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa388c0-7e72-4b86-b692-53d9c53ac30b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg\" (UID: \"4fa388c0-7e72-4b86-b692-53d9c53ac30b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"
Oct 30 18:53:41.735553 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.735526    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa388c0-7e72-4b86-b692-53d9c53ac30b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg\" (UID: \"4fa388c0-7e72-4b86-b692-53d9c53ac30b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"
Oct 30 18:53:41.735675 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.735560    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71072275-42b7-4d5e-9e0c-cac53bf01426-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b\" (UID: \"71072275-42b7-4d5e-9e0c-cac53bf01426\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"
Oct 30 18:53:41.754432 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.754374    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t79qr\" (UniqueName: \"kubernetes.io/projected/4fa388c0-7e72-4b86-b692-53d9c53ac30b-kube-api-access-t79qr\") pod \"payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg\" (UID: \"4fa388c0-7e72-4b86-b692-53d9c53ac30b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"
Oct 30 18:53:41.754639 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.754615    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6m9w\" (UniqueName: \"kubernetes.io/projected/71072275-42b7-4d5e-9e0c-cac53bf01426-kube-api-access-m6m9w\") pod \"payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b\" (UID: \"71072275-42b7-4d5e-9e0c-cac53bf01426\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"
Oct 30 18:53:41.979168 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:41.975693    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"
Oct 30 18:53:42.037206 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.014720    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"
Oct 30 18:53:42.214235 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.213940    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:53:42.380183 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.380147    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-maas-api-tls\") pod \"bf1c2575-55bb-418a-8600-46e682bed18a\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") "
Oct 30 18:53:42.404371 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.380431    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf1c2575-55bb-418a-8600-46e682bed18a-kube-api-access-qn7w9\" (UniqueName: \"kubernetes.io/projected/bf1c2575-55bb-418a-8600-46e682bed18a-kube-api-access-qn7w9\") pod \"bf1c2575-55bb-418a-8600-46e682bed18a\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") "
Oct 30 18:53:42.404371 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.380474    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-metrics-certs\") pod \"bf1c2575-55bb-418a-8600-46e682bed18a\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") "
Oct 30 18:53:42.404371 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.380523    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf1c2575-55bb-418a-8600-46e682bed18a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1c2575-55bb-418a-8600-46e682bed18a-openshift-service-ca\") pod \"bf1c2575-55bb-418a-8600-46e682bed18a\" (UID: \"bf1c2575-55bb-418a-8600-46e682bed18a\") "
Oct 30 18:53:42.405250 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.405187    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf1c2575-55bb-418a-8600-46e682bed18a-openshift-service-ca" pod "bf1c2575-55bb-418a-8600-46e682bed18a" (UID: "bf1c2575-55bb-418a-8600-46e682bed18a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:42.414276 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.412041    4372 generic.go:356] "Generic (PLEG): container finished" podID="bf1c2575-55bb-418a-8600-46e682bed18a" containerID="b848eea24c66522e873aac3b2b6477d7f52e589c0074d0a6fc8ab77ef14bf3e4" exitCode=0
Oct 30 18:53:42.424347 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.424312    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh" event={"ID":"bf1c2575-55bb-418a-8600-46e682bed18a","Type":"ContainerDied","Data":"b848eea24c66522e873aac3b2b6477d7f52e589c0074d0a6fc8ab77ef14bf3e4"}
Oct 30 18:53:42.425578 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.425536    4372 scope.go:122] "RemoveContainer" containerID="b848eea24c66522e873aac3b2b6477d7f52e589c0074d0a6fc8ab77ef14bf3e4"
Oct 30 18:53:42.427095 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.427067    4372 generic.go:356] "Generic (PLEG): container finished" podID="d6009aa5-288e-4eba-903a-7c4ad6a93c44" containerID="cda298eededcb070261bf1d5828ebe0a612a991a0725879a620da48d5ccb82ff" exitCode=0
Oct 30 18:53:42.427218 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.427107    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs" event={"ID":"d6009aa5-288e-4eba-903a-7c4ad6a93c44","Type":"ContainerDied","Data":"cda298eededcb070261bf1d5828ebe0a612a991a0725879a620da48d5ccb82ff"}
Oct 30 18:53:42.430169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.430110    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-maas-api-tls" pod "bf1c2575-55bb-418a-8600-46e682bed18a" (UID: "bf1c2575-55bb-418a-8600-46e682bed18a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:42.430838 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.430806    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1c2575-55bb-418a-8600-46e682bed18a-kube-api-access-qn7w9" pod "bf1c2575-55bb-418a-8600-46e682bed18a" (UID: "bf1c2575-55bb-418a-8600-46e682bed18a"). InnerVolumeSpecName "kube-api-access-qn7w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:42.436393 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.436361    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-metrics-certs" pod "bf1c2575-55bb-418a-8600-46e682bed18a" (UID: "bf1c2575-55bb-418a-8600-46e682bed18a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:42.483031 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.482990    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:42.483031 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.483031    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1c2575-55bb-418a-8600-46e682bed18a-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:42.483363 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.483048    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf1c2575-55bb-418a-8600-46e682bed18a-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:42.483363 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.483061    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn7w9\" (UniqueName: \"kubernetes.io/projected/bf1c2575-55bb-418a-8600-46e682bed18a-kube-api-access-qn7w9\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:42.509099 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.509071    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"
Oct 30 18:53:42.584491 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.584463    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6009aa5-288e-4eba-903a-7c4ad6a93c44-kube-api-access-j64dm\" (UniqueName: \"kubernetes.io/projected/d6009aa5-288e-4eba-903a-7c4ad6a93c44-kube-api-access-j64dm\") pod \"d6009aa5-288e-4eba-903a-7c4ad6a93c44\" (UID: \"d6009aa5-288e-4eba-903a-7c4ad6a93c44\") "
Oct 30 18:53:42.584659 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.584588    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6009aa5-288e-4eba-903a-7c4ad6a93c44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6009aa5-288e-4eba-903a-7c4ad6a93c44-plugins-config-volume\") pod \"d6009aa5-288e-4eba-903a-7c4ad6a93c44\" (UID: \"d6009aa5-288e-4eba-903a-7c4ad6a93c44\") "
Oct 30 18:53:42.584968 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.584944    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6009aa5-288e-4eba-903a-7c4ad6a93c44-plugins-config-volume" pod "d6009aa5-288e-4eba-903a-7c4ad6a93c44" (UID: "d6009aa5-288e-4eba-903a-7c4ad6a93c44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:42.586662 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.586643    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6009aa5-288e-4eba-903a-7c4ad6a93c44-kube-api-access-j64dm" pod "d6009aa5-288e-4eba-903a-7c4ad6a93c44" (UID: "d6009aa5-288e-4eba-903a-7c4ad6a93c44"). InnerVolumeSpecName "kube-api-access-j64dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:42.667534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.667510    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"]
Oct 30 18:53:42.668511 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:53:42.668487    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71072275_42b7_4d5e_9e0c_cac53bf01426.slice/crio-fae602f2e7a4e3dca8cc9da9c2a45158c8861c3fdc3c8aabcff4b2f99b42b41a WatchSource:0}: Error finding container fae602f2e7a4e3dca8cc9da9c2a45158c8861c3fdc3c8aabcff4b2f99b42b41a: Status 404 returned error can't find the container with id fae602f2e7a4e3dca8cc9da9c2a45158c8861c3fdc3c8aabcff4b2f99b42b41a
Oct 30 18:53:42.686364 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.686342    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6009aa5-288e-4eba-903a-7c4ad6a93c44-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:42.686454 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.686365    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j64dm\" (UniqueName: \"kubernetes.io/projected/d6009aa5-288e-4eba-903a-7c4ad6a93c44-kube-api-access-j64dm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:42.735406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:42.735380    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"]
Oct 30 18:53:42.735783 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:53:42.735760    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa388c0_7e72_4b86_b692_53d9c53ac30b.slice/crio-c890ae6bcee9f0a4a3a81e58ff8ca1097774ebca748b3e33afff4c8d4e3e30d6 WatchSource:0}: Error finding container c890ae6bcee9f0a4a3a81e58ff8ca1097774ebca748b3e33afff4c8d4e3e30d6: Status 404 returned error can't find the container with id c890ae6bcee9f0a4a3a81e58ff8ca1097774ebca748b3e33afff4c8d4e3e30d6
Oct 30 18:53:43.436853 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.436814    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b" event={"ID":"71072275-42b7-4d5e-9e0c-cac53bf01426","Type":"ContainerStarted","Data":"b7d83cc279d11901d1bb2f7b632820a3dc4707d15fcbf1acdae4a601c4361dc8"}
Oct 30 18:53:43.437384 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.436881    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b" event={"ID":"71072275-42b7-4d5e-9e0c-cac53bf01426","Type":"ContainerStarted","Data":"fae602f2e7a4e3dca8cc9da9c2a45158c8861c3fdc3c8aabcff4b2f99b42b41a"}
Oct 30 18:53:43.437384 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.436911    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"
Oct 30 18:53:43.438401 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.438384    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"
Oct 30 18:53:43.438484 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.438427    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs" event={"ID":"d6009aa5-288e-4eba-903a-7c4ad6a93c44","Type":"ContainerDied","Data":"11f7356d0bab93dacbb73fbbc25b71c0956b3a487ebf2b720a41166cabf189e0"}
Oct 30 18:53:43.438541 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.438484    4372 scope.go:122] "RemoveContainer" containerID="cda298eededcb070261bf1d5828ebe0a612a991a0725879a620da48d5ccb82ff"
Oct 30 18:53:43.439522 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.439454    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"
Oct 30 18:53:43.439522 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.439495    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh" event={"ID":"bf1c2575-55bb-418a-8600-46e682bed18a","Type":"ContainerDied","Data":"518cb2ef4efff5c893bc89950660bb7967193f643afad8af2c2bcda31a9d346b"}
Oct 30 18:53:43.440964 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.440933    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg" event={"ID":"4fa388c0-7e72-4b86-b692-53d9c53ac30b","Type":"ContainerStarted","Data":"0ac8cafa983c8ded200ba63bdb92da72ac59afa997aa2c9f97a79af04342feac"}
Oct 30 18:53:43.441067 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.440981    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg" event={"ID":"4fa388c0-7e72-4b86-b692-53d9c53ac30b","Type":"ContainerStarted","Data":"c890ae6bcee9f0a4a3a81e58ff8ca1097774ebca748b3e33afff4c8d4e3e30d6"}
Oct 30 18:53:43.441346 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.441279    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"
Oct 30 18:53:43.457170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.457117    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b" podStartSLOduration=2.457104044 podStartE2EDuration="2.457104044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:43.456760658 +0000 UTC m=+1237.325178989" watchObservedRunningTime="2026-10-30 18:53:43.457104044 +0000 UTC m=+1237.325522371"
Oct 30 18:53:43.483642 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.483602    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg" podStartSLOduration=2.48359116 podStartE2EDuration="2.48359116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:43.478532315 +0000 UTC m=+1237.346950643" watchObservedRunningTime="2026-10-30 18:53:43.48359116 +0000 UTC m=+1237.352009491"
Oct 30 18:53:43.509476 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.509445    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"]
Oct 30 18:53:43.511684 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.511651    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecef08-5b94c544b-kvxgs"]
Oct 30 18:53:43.532235 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.532213    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"]
Oct 30 18:53:43.543797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:43.543778    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ecef08-c6447c595-qd6rh"]
Oct 30 18:53:44.323153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:44.314798    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf1c2575-55bb-418a-8600-46e682bed18a" path="/var/lib/kubelet/pods/bf1c2575-55bb-418a-8600-46e682bed18a/volumes"
Oct 30 18:53:44.324323 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:44.324291    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6009aa5-288e-4eba-903a-7c4ad6a93c44" path="/var/lib/kubelet/pods/d6009aa5-288e-4eba-903a-7c4ad6a93c44/volumes"
Oct 30 18:53:48.161382 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:48.161346    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:53:49.079169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:49.075116    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"]
Oct 30 18:53:49.079169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:49.075768    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerName="istio-proxy" containerID="cri-o://0f03dc031131119f1405aefe91eef652f8f19235aca355a55bdde1bc94cc8f9a" gracePeriod=30
Oct 30 18:53:51.356341 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:51.356303    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"]
Oct 30 18:53:51.357183 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:51.357105    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc" podUID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" containerName="istio-proxy" containerID="cri-o://cf11e1148fee1d90cf87dd839e5ded74671015c76bc6e4036a1b3fa8ef543877" gracePeriod=30
Oct 30 18:53:51.748211 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:51.742785    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:53:51.748211 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:51.742874    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:53:52.036012 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.035914    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:52.037612 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.037579    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:52.046348 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.046311    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:52.082842 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.082797    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 18:53:52.121778 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.092550    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:52.297600 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.297469    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"]
Oct 30 18:53:52.299841 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.299818    4372 cpu_manager.go:414] "containerMap: removing container" podUID="bf1c2575-55bb-418a-8600-46e682bed18a" containerName="maas-api"
Oct 30 18:53:52.300071 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.300055    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf1c2575-55bb-418a-8600-46e682bed18a" containerName="maas-api"
Oct 30 18:53:52.300310 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.300295    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d6009aa5-288e-4eba-903a-7c4ad6a93c44" containerName="payload-processing"
Oct 30 18:53:52.300544 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.300448    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6009aa5-288e-4eba-903a-7c4ad6a93c44" containerName="payload-processing"
Oct 30 18:53:52.300955 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.300938    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf1c2575-55bb-418a-8600-46e682bed18a" containerName="maas-api"
Oct 30 18:53:52.301096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.301083    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6009aa5-288e-4eba-903a-7c4ad6a93c44" containerName="payload-processing"
Oct 30 18:53:52.318849 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.318823    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"]
Oct 30 18:53:52.318982 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.318926    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"
Oct 30 18:53:52.324672 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.324646    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-fa7b20\""
Oct 30 18:53:52.342408 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.342388    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"]
Oct 30 18:53:52.346603 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.346587    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"
Oct 30 18:53:52.353336 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.353320    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-fa7b20-dockercfg-jqxhp\""
Oct 30 18:53:52.366271 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.366250    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"]
Oct 30 18:53:52.412097 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.412059    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136d8931-c397-4378-a8eb-c33842cf0776-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54\" (UID: \"136d8931-c397-4378-a8eb-c33842cf0776\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"
Oct 30 18:53:52.412302 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.412116    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtwll\" (UniqueName: \"kubernetes.io/projected/136d8931-c397-4378-a8eb-c33842cf0776-kube-api-access-gtwll\") pod \"payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54\" (UID: \"136d8931-c397-4378-a8eb-c33842cf0776\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"
Oct 30 18:53:52.412403 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.412380    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f7d978-361f-4300-869e-985a9ff73cec-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9\" (UID: \"e2f7d978-361f-4300-869e-985a9ff73cec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"
Oct 30 18:53:52.412745 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.412715    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2vmm\" (UniqueName: \"kubernetes.io/projected/e2f7d978-361f-4300-869e-985a9ff73cec-kube-api-access-c2vmm\") pod \"payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9\" (UID: \"e2f7d978-361f-4300-869e-985a9ff73cec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"
Oct 30 18:53:52.542569 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.541477    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136d8931-c397-4378-a8eb-c33842cf0776-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54\" (UID: \"136d8931-c397-4378-a8eb-c33842cf0776\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"
Oct 30 18:53:52.542902 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.542423    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136d8931-c397-4378-a8eb-c33842cf0776-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54\" (UID: \"136d8931-c397-4378-a8eb-c33842cf0776\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"
Oct 30 18:53:52.543490 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.543103    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtwll\" (UniqueName: \"kubernetes.io/projected/136d8931-c397-4378-a8eb-c33842cf0776-kube-api-access-gtwll\") pod \"payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54\" (UID: \"136d8931-c397-4378-a8eb-c33842cf0776\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"
Oct 30 18:53:52.555049 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.554957    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f7d978-361f-4300-869e-985a9ff73cec-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9\" (UID: \"e2f7d978-361f-4300-869e-985a9ff73cec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"
Oct 30 18:53:52.555243 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.555222    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f7d978-361f-4300-869e-985a9ff73cec-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9\" (UID: \"e2f7d978-361f-4300-869e-985a9ff73cec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"
Oct 30 18:53:52.555571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.555553    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2vmm\" (UniqueName: \"kubernetes.io/projected/e2f7d978-361f-4300-869e-985a9ff73cec-kube-api-access-c2vmm\") pod \"payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9\" (UID: \"e2f7d978-361f-4300-869e-985a9ff73cec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"
Oct 30 18:53:52.579866 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.574355    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtwll\" (UniqueName: \"kubernetes.io/projected/136d8931-c397-4378-a8eb-c33842cf0776-kube-api-access-gtwll\") pod \"payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54\" (UID: \"136d8931-c397-4378-a8eb-c33842cf0776\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"
Oct 30 18:53:52.609946 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.609912    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2vmm\" (UniqueName: \"kubernetes.io/projected/e2f7d978-361f-4300-869e-985a9ff73cec-kube-api-access-c2vmm\") pod \"payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9\" (UID: \"e2f7d978-361f-4300-869e-985a9ff73cec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"
Oct 30 18:53:52.634899 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.634617    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"
Oct 30 18:53:52.660625 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:52.660377    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"
Oct 30 18:53:53.099698 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.099650    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"]
Oct 30 18:53:53.168973 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.168904    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9" event={"ID":"e2f7d978-361f-4300-869e-985a9ff73cec","Type":"ContainerStarted","Data":"f91c88a01fe2dde3d2e3a991486fdf2789b9e7a38ee287ea3fb5b2dea323fd85"}
Oct 30 18:53:53.176608 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.176582    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"]
Oct 30 18:53:53.177021 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:53:53.176996    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136d8931_c397_4378_a8eb_c33842cf0776.slice/crio-7662019938d6710fc2f6952f2663a67aeac4f7f2fbfdf24bc71380fedf13c1c5 WatchSource:0}: Error finding container 7662019938d6710fc2f6952f2663a67aeac4f7f2fbfdf24bc71380fedf13c1c5: Status 404 returned error can't find the container with id 7662019938d6710fc2f6952f2663a67aeac4f7f2fbfdf24bc71380fedf13c1c5
Oct 30 18:53:53.432440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.432373    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"]
Oct 30 18:53:53.439763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.439743    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.446609 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.446584    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d5ab51-openshift-default-dockercfg-pbgxt\""
Oct 30 18:53:53.468813 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.468790    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"]
Oct 30 18:53:53.628153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.626266    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-credential-socket\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.628153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.626416    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-podinfo\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.628153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.626458    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-workload-socket\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.652162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.644072    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-workload-certs\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.652162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.644423    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-envoy\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.652162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.644800    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15bc1bcf-0466-40c4-98af-928b2f3a259d-istiod-ca-cert\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.652162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.645118    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-data\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.652162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.645284    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtpwd\" (UniqueName: \"kubernetes.io/projected/15bc1bcf-0466-40c4-98af-928b2f3a259d-kube-api-access-rtpwd\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.652162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.645357    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-token\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.757234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.750237    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-data\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.757234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.750303    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtpwd\" (UniqueName: \"kubernetes.io/projected/15bc1bcf-0466-40c4-98af-928b2f3a259d-kube-api-access-rtpwd\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.757234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.750334    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-token\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.757234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.750411    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-credential-socket\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.757234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.750481    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-podinfo\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.757234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.750511    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-workload-socket\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.757234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.750564    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-workload-certs\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.757234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.750634    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-envoy\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.757234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.750722    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15bc1bcf-0466-40c4-98af-928b2f3a259d-istiod-ca-cert\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.757234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.751732    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15bc1bcf-0466-40c4-98af-928b2f3a259d-istiod-ca-cert\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.757234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.752204    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-data\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.757234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.752970    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-credential-socket\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.777151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.760516    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-workload-certs\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.777151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.760751    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-workload-socket\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.777151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.771766    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-envoy\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.836456 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.836390    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-podinfo\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.878107 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.878042    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtpwd\" (UniqueName: \"kubernetes.io/projected/15bc1bcf-0466-40c4-98af-928b2f3a259d-kube-api-access-rtpwd\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:53.893158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:53.889855    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-token\") pod \"e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") " pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:54.054407 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.054327    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:54.178811 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.178536    4372 generic.go:356] "Generic (PLEG): container finished" podID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerID="0f03dc031131119f1405aefe91eef652f8f19235aca355a55bdde1bc94cc8f9a" exitCode=0
Oct 30 18:53:54.178811 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.178634    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" event={"ID":"2313cd49-c455-47e7-94df-091ce8c2edf9","Type":"ContainerDied","Data":"0f03dc031131119f1405aefe91eef652f8f19235aca355a55bdde1bc94cc8f9a"}
Oct 30 18:53:54.178811 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.178693    4372 scope.go:122] "RemoveContainer" containerID="2adf405b22604db675e0895e8d6dd795a17d4de6976b9902cef1ed0ec9bf3d2e"
Oct 30 18:53:54.182816 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.182421    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9" event={"ID":"e2f7d978-361f-4300-869e-985a9ff73cec","Type":"ContainerStarted","Data":"dc1c6ce3b8f4bfe5768733417af250c61dcb251fdbfafd860d2619b58dca5c31"}
Oct 30 18:53:54.183057 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.183009    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"
Oct 30 18:53:54.187890 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.187403    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54" event={"ID":"136d8931-c397-4378-a8eb-c33842cf0776","Type":"ContainerStarted","Data":"3de0c7b54a6948eb777485a6b5f946f9ed869763861349908edc8c023400138b"}
Oct 30 18:53:54.187890 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.187451    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54" event={"ID":"136d8931-c397-4378-a8eb-c33842cf0776","Type":"ContainerStarted","Data":"7662019938d6710fc2f6952f2663a67aeac4f7f2fbfdf24bc71380fedf13c1c5"}
Oct 30 18:53:54.189496 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.189446    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"
Oct 30 18:53:54.233028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.232967    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9" podStartSLOduration=2.232949594 podStartE2EDuration="2.232949594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:54.210415139 +0000 UTC m=+1248.078833471" watchObservedRunningTime="2026-10-30 18:53:54.232949594 +0000 UTC m=+1248.101367924"
Oct 30 18:53:54.233997 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.233955    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54" podStartSLOduration=2.233940451 podStartE2EDuration="2.233940451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:54.229556764 +0000 UTC m=+1248.097975096" watchObservedRunningTime="2026-10-30 18:53:54.233940451 +0000 UTC m=+1248.102358782"
Oct 30 18:53:54.400845 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.399960    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"]
Oct 30 18:53:54.492406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.492374    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"
Oct 30 18:53:54.532170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.483958    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:54.532170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.495565    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:54.532170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.495663    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:54.532771 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.532743    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"
Oct 30 18:53:54.870504 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.870477    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:53:54.916869 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.916628    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-token\" (UniqueName: \"kubernetes.io/projected/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-token\") pod \"2313cd49-c455-47e7-94df-091ce8c2edf9\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") "
Oct 30 18:53:54.917154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.917115    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-data\") pod \"2313cd49-c455-47e7-94df-091ce8c2edf9\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") "
Oct 30 18:53:54.917307 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.917291    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-podinfo\") pod \"2313cd49-c455-47e7-94df-091ce8c2edf9\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") "
Oct 30 18:53:54.917475 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.917449    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-data" pod "2313cd49-c455-47e7-94df-091ce8c2edf9" (UID: "2313cd49-c455-47e7-94df-091ce8c2edf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:54.917655 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.917635    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-workload-socket\") pod \"2313cd49-c455-47e7-94df-091ce8c2edf9\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") "
Oct 30 18:53:54.917877 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.917860    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-workload-certs\") pod \"2313cd49-c455-47e7-94df-091ce8c2edf9\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") "
Oct 30 18:53:54.918036 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.918011    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-envoy\") pod \"2313cd49-c455-47e7-94df-091ce8c2edf9\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") "
Oct 30 18:53:54.918178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.917868    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-workload-socket" pod "2313cd49-c455-47e7-94df-091ce8c2edf9" (UID: "2313cd49-c455-47e7-94df-091ce8c2edf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:54.918178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.918019    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-workload-certs" pod "2313cd49-c455-47e7-94df-091ce8c2edf9" (UID: "2313cd49-c455-47e7-94df-091ce8c2edf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:54.918505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.918486    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2313cd49-c455-47e7-94df-091ce8c2edf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2313cd49-c455-47e7-94df-091ce8c2edf9-istiod-ca-cert\") pod \"2313cd49-c455-47e7-94df-091ce8c2edf9\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") "
Oct 30 18:53:54.918638 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.918621    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-credential-socket\") pod \"2313cd49-c455-47e7-94df-091ce8c2edf9\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") "
Oct 30 18:53:54.918752 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.918736    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2313cd49-c455-47e7-94df-091ce8c2edf9-kube-api-access-l2tps\" (UniqueName: \"kubernetes.io/projected/2313cd49-c455-47e7-94df-091ce8c2edf9-kube-api-access-l2tps\") pod \"2313cd49-c455-47e7-94df-091ce8c2edf9\" (UID: \"2313cd49-c455-47e7-94df-091ce8c2edf9\") "
Oct 30 18:53:54.919693 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.918969    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-credential-socket" pod "2313cd49-c455-47e7-94df-091ce8c2edf9" (UID: "2313cd49-c455-47e7-94df-091ce8c2edf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:54.919693 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.919457    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:54.919693 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.919479    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:54.919693 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.919493    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:54.919693 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.919506    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:54.919693 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.919665    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2313cd49-c455-47e7-94df-091ce8c2edf9-istiod-ca-cert" pod "2313cd49-c455-47e7-94df-091ce8c2edf9" (UID: "2313cd49-c455-47e7-94df-091ce8c2edf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:54.920280 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.920254    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-token" pod "2313cd49-c455-47e7-94df-091ce8c2edf9" (UID: "2313cd49-c455-47e7-94df-091ce8c2edf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:54.920692 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.920670    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-envoy" pod "2313cd49-c455-47e7-94df-091ce8c2edf9" (UID: "2313cd49-c455-47e7-94df-091ce8c2edf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:54.921999 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.921973    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2313cd49-c455-47e7-94df-091ce8c2edf9-kube-api-access-l2tps" pod "2313cd49-c455-47e7-94df-091ce8c2edf9" (UID: "2313cd49-c455-47e7-94df-091ce8c2edf9"). InnerVolumeSpecName "kube-api-access-l2tps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:54.922922 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:54.922880    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-podinfo" pod "2313cd49-c455-47e7-94df-091ce8c2edf9" (UID: "2313cd49-c455-47e7-94df-091ce8c2edf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:53:55.020446 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:55.020414    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:55.020446 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:55.020444    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2313cd49-c455-47e7-94df-091ce8c2edf9-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:55.020446 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:55.020454    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2tps\" (UniqueName: \"kubernetes.io/projected/2313cd49-c455-47e7-94df-091ce8c2edf9-kube-api-access-l2tps\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:55.020685 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:55.020464    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:55.020685 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:55.020472    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2313cd49-c455-47e7-94df-091ce8c2edf9-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:55.210507 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:55.210431    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"
Oct 30 18:53:55.210507 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:55.210467    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85" event={"ID":"2313cd49-c455-47e7-94df-091ce8c2edf9","Type":"ContainerDied","Data":"c329fe6d7baed360eff04ff3d280dc4317219961c11843abf9d88b63a0d533a5"}
Oct 30 18:53:55.210753 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:55.210539    4372 scope.go:122] "RemoveContainer" containerID="0f03dc031131119f1405aefe91eef652f8f19235aca355a55bdde1bc94cc8f9a"
Oct 30 18:53:55.212310 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:55.212285    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk" event={"ID":"15bc1bcf-0466-40c4-98af-928b2f3a259d","Type":"ContainerStarted","Data":"77ef934c9b05baf0b75ac32cc792f9209ad4805e53e7007016ece1e9d6f25808"}
Oct 30 18:53:55.212414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:55.212337    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk" event={"ID":"15bc1bcf-0466-40c4-98af-928b2f3a259d","Type":"ContainerStarted","Data":"9f8b2de42e70cbb1efb95783cf8ca866f7bbfb023c29f8fff26d31dfe3dc5e2e"}
Oct 30 18:53:55.243294 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:55.243250    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk" podStartSLOduration=2.243238734 podStartE2EDuration="2.243238734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:55.239772462 +0000 UTC m=+1249.108190795" watchObservedRunningTime="2026-10-30 18:53:55.243238734 +0000 UTC m=+1249.111657063"
Oct 30 18:53:55.258836 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:55.258805    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"]
Oct 30 18:53:55.265384 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:55.265360    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8fd4ed-openshift-default-7bf9bc46c9-pqj85"]
Oct 30 18:53:56.055085 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.055051    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:56.060440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.060413    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:56.220380 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.220347    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:56.221248 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.221227    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:53:56.270977 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.270949    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" path="/var/lib/kubelet/pods/2313cd49-c455-47e7-94df-091ce8c2edf9/volumes"
Oct 30 18:53:56.747883 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.747856    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:53:56.851904 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.851874    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-podinfo\") pod \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") "
Oct 30 18:53:56.852064 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.851959    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-workload-certs\") pod \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") "
Oct 30 18:53:56.852064 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.852026    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-envoy\") pod \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") "
Oct 30 18:53:56.852064 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.852057    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-token\" (UniqueName: \"kubernetes.io/projected/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-token\") pod \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") "
Oct 30 18:53:56.852229 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.852218    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-data\") pod \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") "
Oct 30 18:53:56.852292 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.852258    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-credential-socket\") pod \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") "
Oct 30 18:53:56.852346 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.852300    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-kube-api-access-hln4t\" (UniqueName: \"kubernetes.io/projected/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-kube-api-access-hln4t\") pod \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") "
Oct 30 18:53:56.852346 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.852321    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-workload-certs" pod "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" (UID: "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:56.852449 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.852333    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-workload-socket\") pod \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") "
Oct 30 18:53:56.852572 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.852543    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-credential-socket" pod "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" (UID: "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:56.852639 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.852556    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-workload-socket" pod "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" (UID: "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:56.852725 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.852699    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-data" pod "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" (UID: "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:56.852785 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.852454    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istiod-ca-cert\") pod \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\" (UID: \"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04\") "
Oct 30 18:53:56.852945 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.852922    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istiod-ca-cert" pod "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" (UID: "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:56.853410 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.853389    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:56.853507 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.853414    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:56.853507 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.853431    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:56.853507 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.853445    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:56.853507 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.853458    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:56.854466 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.854442    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-envoy" pod "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" (UID: "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:56.855106 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.855086    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-podinfo" pod "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" (UID: "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:53:56.855354 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.855318    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-kube-api-access-hln4t" pod "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" (UID: "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04"). InnerVolumeSpecName "kube-api-access-hln4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:56.855922 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.855899    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-token" pod "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" (UID: "bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:56.955346 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.955312    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:56.955346 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.955347    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:56.955732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.955362    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:56.955732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:56.955376    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hln4t\" (UniqueName: \"kubernetes.io/projected/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04-kube-api-access-hln4t\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:57.051038 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.050960    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"]
Oct 30 18:53:57.051848 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.051830    4372 cpu_manager.go:414] "containerMap: removing container" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerName="istio-proxy"
Oct 30 18:53:57.051893 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.051850    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerName="istio-proxy"
Oct 30 18:53:57.051893 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.051890    4372 cpu_manager.go:414] "containerMap: removing container" podUID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" containerName="istio-proxy"
Oct 30 18:53:57.051971 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.051898    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" containerName="istio-proxy"
Oct 30 18:53:57.051971 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.051913    4372 cpu_manager.go:414] "containerMap: removing container" podUID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" containerName="istio-proxy"
Oct 30 18:53:57.051971 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.051921    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" containerName="istio-proxy"
Oct 30 18:53:57.052067 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.052028    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerName="istio-proxy"
Oct 30 18:53:57.052067 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.052045    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" containerName="istio-proxy"
Oct 30 18:53:57.052067 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.052054    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerName="istio-proxy"
Oct 30 18:53:57.052067 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.052065    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" containerName="istio-proxy"
Oct 30 18:53:57.056265 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.056247    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"
Oct 30 18:53:57.064471 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.064450    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f2ceb5\""
Oct 30 18:53:57.064801 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.064765    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f2ceb5-dockercfg-hw7l5\""
Oct 30 18:53:57.070189 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.070167    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"]
Oct 30 18:53:57.087937 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.087909    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"]
Oct 30 18:53:57.088549 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.088528    4372 cpu_manager.go:414] "containerMap: removing container" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerName="istio-proxy"
Oct 30 18:53:57.088549 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.088546    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2313cd49-c455-47e7-94df-091ce8c2edf9" containerName="istio-proxy"
Oct 30 18:53:57.092269 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.092251    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"
Oct 30 18:53:57.106791 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.106738    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"]
Oct 30 18:53:57.253155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.212231    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c35fda7-eeab-45c3-9fdc-fc00bc12f141-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99\" (UID: \"9c35fda7-eeab-45c3-9fdc-fc00bc12f141\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"
Oct 30 18:53:57.253155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.212340    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgfp9\" (UniqueName: \"kubernetes.io/projected/9c35fda7-eeab-45c3-9fdc-fc00bc12f141-kube-api-access-zgfp9\") pod \"payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99\" (UID: \"9c35fda7-eeab-45c3-9fdc-fc00bc12f141\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"
Oct 30 18:53:57.253155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.212484    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk25f\" (UniqueName: \"kubernetes.io/projected/1d5c2659-c280-40fb-b2da-bf7672b55833-kube-api-access-nk25f\") pod \"payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98\" (UID: \"1d5c2659-c280-40fb-b2da-bf7672b55833\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"
Oct 30 18:53:57.253155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.212511    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5c2659-c280-40fb-b2da-bf7672b55833-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98\" (UID: \"1d5c2659-c280-40fb-b2da-bf7672b55833\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"
Oct 30 18:53:57.326199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.320058    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c35fda7-eeab-45c3-9fdc-fc00bc12f141-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99\" (UID: \"9c35fda7-eeab-45c3-9fdc-fc00bc12f141\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"
Oct 30 18:53:57.353190 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.327302    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c35fda7-eeab-45c3-9fdc-fc00bc12f141-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99\" (UID: \"9c35fda7-eeab-45c3-9fdc-fc00bc12f141\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"
Oct 30 18:53:57.353579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.353554    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgfp9\" (UniqueName: \"kubernetes.io/projected/9c35fda7-eeab-45c3-9fdc-fc00bc12f141-kube-api-access-zgfp9\") pod \"payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99\" (UID: \"9c35fda7-eeab-45c3-9fdc-fc00bc12f141\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"
Oct 30 18:53:57.358159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.354264    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk25f\" (UniqueName: \"kubernetes.io/projected/1d5c2659-c280-40fb-b2da-bf7672b55833-kube-api-access-nk25f\") pod \"payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98\" (UID: \"1d5c2659-c280-40fb-b2da-bf7672b55833\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"
Oct 30 18:53:57.362258 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.362220    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5c2659-c280-40fb-b2da-bf7672b55833-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98\" (UID: \"1d5c2659-c280-40fb-b2da-bf7672b55833\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"
Oct 30 18:53:57.385194 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.367708    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5c2659-c280-40fb-b2da-bf7672b55833-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98\" (UID: \"1d5c2659-c280-40fb-b2da-bf7672b55833\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"
Oct 30 18:53:57.399846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.399739    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk25f\" (UniqueName: \"kubernetes.io/projected/1d5c2659-c280-40fb-b2da-bf7672b55833-kube-api-access-nk25f\") pod \"payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98\" (UID: \"1d5c2659-c280-40fb-b2da-bf7672b55833\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"
Oct 30 18:53:57.410860 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.410613    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"
Oct 30 18:53:57.422058 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.420103    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgfp9\" (UniqueName: \"kubernetes.io/projected/9c35fda7-eeab-45c3-9fdc-fc00bc12f141-kube-api-access-zgfp9\") pod \"payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99\" (UID: \"9c35fda7-eeab-45c3-9fdc-fc00bc12f141\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"
Oct 30 18:53:57.425571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.422621    4372 generic.go:356] "Generic (PLEG): container finished" podID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" containerID="cf11e1148fee1d90cf87dd839e5ded74671015c76bc6e4036a1b3fa8ef543877" exitCode=0
Oct 30 18:53:57.425571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.424031    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"
Oct 30 18:53:57.425571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.424643    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc" event={"ID":"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04","Type":"ContainerDied","Data":"cf11e1148fee1d90cf87dd839e5ded74671015c76bc6e4036a1b3fa8ef543877"}
Oct 30 18:53:57.425571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.424699    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc" event={"ID":"bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04","Type":"ContainerDied","Data":"0d7f2ea94fda9e61c7a3fa459f637fe8db41968ff6757ae49e3550ea07181e38"}
Oct 30 18:53:57.425571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.424729    4372 scope.go:122] "RemoveContainer" containerID="cf11e1148fee1d90cf87dd839e5ded74671015c76bc6e4036a1b3fa8ef543877"
Oct 30 18:53:57.461383 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.461367    4372 scope.go:122] "RemoveContainer" containerID="0fb14a8ccf447af93953cf2b5ac08e30bb33616ce4b9d0e69f21b9e723567e34"
Oct 30 18:53:57.484290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.484264    4372 scope.go:122] "RemoveContainer" containerID="cf11e1148fee1d90cf87dd839e5ded74671015c76bc6e4036a1b3fa8ef543877"
Oct 30 18:53:57.484616 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:53:57.484560    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf11e1148fee1d90cf87dd839e5ded74671015c76bc6e4036a1b3fa8ef543877\": container with ID starting with cf11e1148fee1d90cf87dd839e5ded74671015c76bc6e4036a1b3fa8ef543877 not found: ID does not exist" containerID="cf11e1148fee1d90cf87dd839e5ded74671015c76bc6e4036a1b3fa8ef543877"
Oct 30 18:53:57.484616 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.484602    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf11e1148fee1d90cf87dd839e5ded74671015c76bc6e4036a1b3fa8ef543877"} err="failed to get container status \"cf11e1148fee1d90cf87dd839e5ded74671015c76bc6e4036a1b3fa8ef543877\": rpc error: code = NotFound desc = could not find container \"cf11e1148fee1d90cf87dd839e5ded74671015c76bc6e4036a1b3fa8ef543877\": container with ID starting with cf11e1148fee1d90cf87dd839e5ded74671015c76bc6e4036a1b3fa8ef543877 not found: ID does not exist"
Oct 30 18:53:57.484797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.484630    4372 scope.go:122] "RemoveContainer" containerID="0fb14a8ccf447af93953cf2b5ac08e30bb33616ce4b9d0e69f21b9e723567e34"
Oct 30 18:53:57.484966 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:53:57.484945    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb14a8ccf447af93953cf2b5ac08e30bb33616ce4b9d0e69f21b9e723567e34\": container with ID starting with 0fb14a8ccf447af93953cf2b5ac08e30bb33616ce4b9d0e69f21b9e723567e34 not found: ID does not exist" containerID="0fb14a8ccf447af93953cf2b5ac08e30bb33616ce4b9d0e69f21b9e723567e34"
Oct 30 18:53:57.485023 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.484980    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb14a8ccf447af93953cf2b5ac08e30bb33616ce4b9d0e69f21b9e723567e34"} err="failed to get container status \"0fb14a8ccf447af93953cf2b5ac08e30bb33616ce4b9d0e69f21b9e723567e34\": rpc error: code = NotFound desc = could not find container \"0fb14a8ccf447af93953cf2b5ac08e30bb33616ce4b9d0e69f21b9e723567e34\": container with ID starting with 0fb14a8ccf447af93953cf2b5ac08e30bb33616ce4b9d0e69f21b9e723567e34 not found: ID does not exist"
Oct 30 18:53:57.508868 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.508843    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"]
Oct 30 18:53:57.517465 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.516940    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ecef08-openshift-default-69799457f-5llxc"]
Oct 30 18:53:57.673066 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:57.672985    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"
Oct 30 18:53:57.705313 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:53:57.705275    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:53:58.030346 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:58.027202    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"]
Oct 30 18:53:58.220154 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:53:58.220096    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c35fda7_eeab_45c3_9fdc_fc00bc12f141.slice/crio-c1a919c6d565122241c02d1308a7abe8536fd5c9da7bd19303910765e1dd71da WatchSource:0}: Error finding container c1a919c6d565122241c02d1308a7abe8536fd5c9da7bd19303910765e1dd71da: Status 404 returned error can't find the container with id c1a919c6d565122241c02d1308a7abe8536fd5c9da7bd19303910765e1dd71da
Oct 30 18:53:58.221090 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:58.221062    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"]
Oct 30 18:53:58.269898 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:58.269867    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04" path="/var/lib/kubelet/pods/bdbe58a7-df0d-4e2b-a4bf-494ebc67ba04/volumes"
Oct 30 18:53:58.439180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:58.439120    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98" event={"ID":"1d5c2659-c280-40fb-b2da-bf7672b55833","Type":"ContainerStarted","Data":"7b9d4bf8b57eb9485642fdb601f6a81b27ee9ed2ad02b37ba91078ea7fac5fed"}
Oct 30 18:53:58.439381 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:58.439207    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98" event={"ID":"1d5c2659-c280-40fb-b2da-bf7672b55833","Type":"ContainerStarted","Data":"720f4f35c2c2122f51fc6aa082d16c565447f7b916e648dfd84e48f746c3fa55"}
Oct 30 18:53:58.439381 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:58.439258    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"
Oct 30 18:53:58.442363 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:58.442337    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99" event={"ID":"9c35fda7-eeab-45c3-9fdc-fc00bc12f141","Type":"ContainerStarted","Data":"a991aa4040b1633a35b5e230260cc5378751cc11619b6b68be9d22a2c0d7df45"}
Oct 30 18:53:58.442510 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:58.442377    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99" event={"ID":"9c35fda7-eeab-45c3-9fdc-fc00bc12f141","Type":"ContainerStarted","Data":"c1a919c6d565122241c02d1308a7abe8536fd5c9da7bd19303910765e1dd71da"}
Oct 30 18:53:58.442510 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:58.442442    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"
Oct 30 18:53:58.465632 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:58.465564    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98" podStartSLOduration=1.465546534 podStartE2EDuration="1.465546534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:58.46144453 +0000 UTC m=+1252.329862862" watchObservedRunningTime="2026-10-30 18:53:58.465546534 +0000 UTC m=+1252.333964866"
Oct 30 18:53:58.485784 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:58.485729    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99" podStartSLOduration=1.48571528 podStartE2EDuration="1.48571528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:58.482241412 +0000 UTC m=+1252.350659752" watchObservedRunningTime="2026-10-30 18:53:58.48571528 +0000 UTC m=+1252.354133610"
Oct 30 18:53:58.995939 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:58.995899    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"]
Oct 30 18:53:59.000331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:59.000308    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"
Oct 30 18:53:59.010377 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:59.010353    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d5ab51\""
Oct 30 18:53:59.011407 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:59.011388    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d5ab51-dockercfg-lz8xd\""
Oct 30 18:53:59.037209 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:59.037182    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"]
Oct 30 18:53:59.201152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:59.200712    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55e51e05-f986-45b9-836d-6339a685318d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95\" (UID: \"55e51e05-f986-45b9-836d-6339a685318d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"
Oct 30 18:53:59.201152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:59.200918    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5slpt\" (UniqueName: \"kubernetes.io/projected/55e51e05-f986-45b9-836d-6339a685318d-kube-api-access-5slpt\") pod \"payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95\" (UID: \"55e51e05-f986-45b9-836d-6339a685318d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"
Oct 30 18:53:59.347175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:59.340432    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55e51e05-f986-45b9-836d-6339a685318d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95\" (UID: \"55e51e05-f986-45b9-836d-6339a685318d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"
Oct 30 18:53:59.347175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:59.340641    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5slpt\" (UniqueName: \"kubernetes.io/projected/55e51e05-f986-45b9-836d-6339a685318d-kube-api-access-5slpt\") pod \"payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95\" (UID: \"55e51e05-f986-45b9-836d-6339a685318d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"
Oct 30 18:53:59.347175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:59.341870    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55e51e05-f986-45b9-836d-6339a685318d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95\" (UID: \"55e51e05-f986-45b9-836d-6339a685318d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"
Oct 30 18:53:59.364167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:59.364090    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5slpt\" (UniqueName: \"kubernetes.io/projected/55e51e05-f986-45b9-836d-6339a685318d-kube-api-access-5slpt\") pod \"payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95\" (UID: \"55e51e05-f986-45b9-836d-6339a685318d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"
Oct 30 18:53:59.615226 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:59.615111    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"
Oct 30 18:53:59.922100 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:53:59.922052    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"]
Oct 30 18:53:59.922419 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:53:59.922377    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e51e05_f986_45b9_836d_6339a685318d.slice/crio-d327654a3ec4c1f98566412af1062ce4c5712f6e6b231ce9b3939a1f49d0324d WatchSource:0}: Error finding container d327654a3ec4c1f98566412af1062ce4c5712f6e6b231ce9b3939a1f49d0324d: Status 404 returned error can't find the container with id d327654a3ec4c1f98566412af1062ce4c5712f6e6b231ce9b3939a1f49d0324d
Oct 30 18:54:00.478219 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:00.478172    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95" event={"ID":"55e51e05-f986-45b9-836d-6339a685318d","Type":"ContainerStarted","Data":"0056289b238e757598abbe59e9f05cab3d0e18db0a96a3f933477f50dc750b5f"}
Oct 30 18:54:00.478723 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:00.478234    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95" event={"ID":"55e51e05-f986-45b9-836d-6339a685318d","Type":"ContainerStarted","Data":"d327654a3ec4c1f98566412af1062ce4c5712f6e6b231ce9b3939a1f49d0324d"}
Oct 30 18:54:00.478723 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:00.478264    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"
Oct 30 18:54:00.500833 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:00.500785    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95" podStartSLOduration=2.5007676009999997 podStartE2EDuration="2.500767601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:00.497551786 +0000 UTC m=+1254.365970116" watchObservedRunningTime="2026-10-30 18:54:00.500767601 +0000 UTC m=+1254.369185933"
Oct 30 18:54:01.486646 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:01.486612    4372 generic.go:356] "Generic (PLEG): container finished" podID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" containerID="ad245b65e6a015a18a9ba10eafe03212a0977da46bee1264b343f742b4563490" exitCode=0
Oct 30 18:54:01.487095 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:01.486683    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74" event={"ID":"57b527cf-ac31-44b7-8ff0-6e40ee60ce32","Type":"ContainerDied","Data":"ad245b65e6a015a18a9ba10eafe03212a0977da46bee1264b343f742b4563490"}
Oct 30 18:54:01.487361 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:01.487343    4372 scope.go:122] "RemoveContainer" containerID="ad245b65e6a015a18a9ba10eafe03212a0977da46bee1264b343f742b4563490"
Oct 30 18:54:01.488898 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:01.488873    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:54:01.488953 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:01.488939    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:54:01.488988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:01.488971    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:54:02.514175 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:02.514104    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74" event={"ID":"57b527cf-ac31-44b7-8ff0-6e40ee60ce32","Type":"ContainerStarted","Data":"e6ba8b07a6fe0acca60eeedc43c912190fd89c1b21b5e3e1abb5467dc280801d"}
Oct 30 18:54:02.963437 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:02.963404    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg_4fa388c0-7e72-4b86-b692-53d9c53ac30b/payload-processing/0.log"
Oct 30 18:54:03.316050 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:03.315963    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:54:03.317277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:03.317252    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 18:54:03.317414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:03.317305    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74" podUID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 18:54:04.315644 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.315611    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 18:54:04.316063 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.315664    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74" podUID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 18:54:04.539863 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.539826    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"]
Oct 30 18:54:04.544078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.544054    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:04.556072 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.556020    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-f2ceb5-serving-cert\""
Oct 30 18:54:04.566084 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.563804    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"]
Oct 30 18:54:04.606178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.577235    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-metrics-certs\") pod \"maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:04.606178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.577272    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h42tq\" (UniqueName: \"kubernetes.io/projected/16453557-85f1-483b-8439-449950e8852d-kube-api-access-h42tq\") pod \"maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:04.606178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.577446    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16453557-85f1-483b-8439-449950e8852d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:04.606178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.577478    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:04.693468 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.683838    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16453557-85f1-483b-8439-449950e8852d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:04.693468 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.683924    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:04.693468 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.684146    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-metrics-certs\") pod \"maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:04.693468 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.684175    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h42tq\" (UniqueName: \"kubernetes.io/projected/16453557-85f1-483b-8439-449950e8852d-kube-api-access-h42tq\") pod \"maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:04.693468 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.685382    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16453557-85f1-483b-8439-449950e8852d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:04.693468 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:54:04.685491    4372 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-serving-cert: secret "maas-api-e2e-praxis-a-f2ceb5-serving-cert" not found
Oct 30 18:54:04.693468 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:54:04.685548    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-maas-api-tls podName:16453557-85f1-483b-8439-449950e8852d nodeName:}" failed. No retries permitted until 2026-10-30 18:54:05.185529187 +0000 UTC m=+1259.053947496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-maas-api-tls") pod "maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng" (UID: "16453557-85f1-483b-8439-449950e8852d") : secret "maas-api-e2e-praxis-a-f2ceb5-serving-cert" not found
Oct 30 18:54:04.795152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.779944    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-metrics-certs\") pod \"maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:04.795152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.780726    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h42tq\" (UniqueName: \"kubernetes.io/projected/16453557-85f1-483b-8439-449950e8852d-kube-api-access-h42tq\") pod \"maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:04.923221 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.923184    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"]
Oct 30 18:54:04.929692 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.929660    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"]
Oct 30 18:54:04.929846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.929745    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:04.939801 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.939781    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-9ceefc\"/\"default-dockercfg-ndtgz\""
Oct 30 18:54:04.939925 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.939859    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-9ceefc\"/\"kube-root-ca.crt\""
Oct 30 18:54:04.941674 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.941655    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-9ceefc\"/\"openshift-service-ca.crt\""
Oct 30 18:54:04.941782 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:04.941742    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-9ceefc\"/\"ipp-route-9ceefc-kserve-self-signed-certs\""
Oct 30 18:54:05.029154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.029063    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-dshm\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.029154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.029145    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-model-cache\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.029429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.029213    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68833d6-91b3-4c78-9115-aac5f70dbba8-tls-certs\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.029429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.029248    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-home\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.029429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.029324    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-tmp-dir\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.029625 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.029533    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wk5\" (UniqueName: \"kubernetes.io/projected/a68833d6-91b3-4c78-9115-aac5f70dbba8-kube-api-access-s7wk5\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.133147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.132331    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-tmp-dir\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.133147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.132540    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wk5\" (UniqueName: \"kubernetes.io/projected/a68833d6-91b3-4c78-9115-aac5f70dbba8-kube-api-access-s7wk5\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.133147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.132704    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-dshm\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.133147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.132747    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-model-cache\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.133147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.132801    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68833d6-91b3-4c78-9115-aac5f70dbba8-tls-certs\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.133147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.132833    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-home\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.143342 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.142615    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-model-cache\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.143342 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.142881    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-tmp-dir\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.156234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.143991    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-home\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.186110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.186034    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-dshm\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.187198 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.187175    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68833d6-91b3-4c78-9115-aac5f70dbba8-tls-certs\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.219007 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.218395    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wk5\" (UniqueName: \"kubernetes.io/projected/a68833d6-91b3-4c78-9115-aac5f70dbba8-kube-api-access-s7wk5\") pod \"ipp-route-9ceefc-kserve-6c55f4b797-gzzdc\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") " pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.237150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.233627    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"
Oct 30 18:54:05.237150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.236368    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:05.237771 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.237747    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"
Oct 30 18:54:05.259153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.259075    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:05.294687 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.286769    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:05.334154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.322231    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 18:54:05.334154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.322386    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74" podUID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 18:54:05.525713 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:05.525622    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:06.057012 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.056977    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:54:06.071956 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.071932    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"]
Oct 30 18:54:06.072987 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:54:06.072941    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda68833d6_91b3_4c78_9115_aac5f70dbba8.slice/crio-2daf6e2f80537d6be52ff645eb69841853e0bba6eef661eb106440196ba3db8d WatchSource:0}: Error finding container 2daf6e2f80537d6be52ff645eb69841853e0bba6eef661eb106440196ba3db8d: Status 404 returned error can't find the container with id 2daf6e2f80537d6be52ff645eb69841853e0bba6eef661eb106440196ba3db8d
Oct 30 18:54:06.161889 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.161862    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"]
Oct 30 18:54:06.163557 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:54:06.163533    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16453557_85f1_483b_8439_449950e8852d.slice/crio-f3943aa0157bd750ee92d9f067ea1d0672e1362224d63ec87282d8a2eb4662ce WatchSource:0}: Error finding container f3943aa0157bd750ee92d9f067ea1d0672e1362224d63ec87282d8a2eb4662ce: Status 404 returned error can't find the container with id f3943aa0157bd750ee92d9f067ea1d0672e1362224d63ec87282d8a2eb4662ce
Oct 30 18:54:06.165526 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.165506    4372 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:54:06.316469 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.316437    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 18:54:06.316641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.316490    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74" podUID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 18:54:06.562249 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.562163    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc" event={"ID":"a68833d6-91b3-4c78-9115-aac5f70dbba8","Type":"ContainerStarted","Data":"e6ea3dc8d6a1eace39fb14dae0cc25a09621bf537b5c920ea16ccbb64f8b0deb"}
Oct 30 18:54:06.562249 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.562207    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc" event={"ID":"a68833d6-91b3-4c78-9115-aac5f70dbba8","Type":"ContainerStarted","Data":"2daf6e2f80537d6be52ff645eb69841853e0bba6eef661eb106440196ba3db8d"}
Oct 30 18:54:06.563743 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.563713    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng" event={"ID":"16453557-85f1-483b-8439-449950e8852d","Type":"ContainerStarted","Data":"f3943aa0157bd750ee92d9f067ea1d0672e1362224d63ec87282d8a2eb4662ce"}
Oct 30 18:54:06.620934 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.620890    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc" podStartSLOduration=2.620871995 podStartE2EDuration="2.620871995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:06.600384944 +0000 UTC m=+1260.468803275" watchObservedRunningTime="2026-10-30 18:54:06.620871995 +0000 UTC m=+1260.489290325"
Oct 30 18:54:06.622956 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.622933    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"]
Oct 30 18:54:06.628166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.628148    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:06.649442 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.649416    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d5ab51-serving-cert\""
Oct 30 18:54:06.663095 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.663066    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"]
Oct 30 18:54:06.695435 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.686363    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6846d27-0053-4c6b-810a-1eab95f42638-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:06.695435 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.686458    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6846d27-0053-4c6b-810a-1eab95f42638-metrics-certs\") pod \"maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:06.695435 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.686543    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6846d27-0053-4c6b-810a-1eab95f42638-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:06.695435 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.686625    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdxsz\" (UniqueName: \"kubernetes.io/projected/f6846d27-0053-4c6b-810a-1eab95f42638-kube-api-access-wdxsz\") pod \"maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:06.805154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.802332    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdxsz\" (UniqueName: \"kubernetes.io/projected/f6846d27-0053-4c6b-810a-1eab95f42638-kube-api-access-wdxsz\") pod \"maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:06.805154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.802579    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6846d27-0053-4c6b-810a-1eab95f42638-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:06.805154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.802669    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6846d27-0053-4c6b-810a-1eab95f42638-metrics-certs\") pod \"maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:06.805154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.802859    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6846d27-0053-4c6b-810a-1eab95f42638-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:06.805154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.803856    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6846d27-0053-4c6b-810a-1eab95f42638-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:06.874505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.871985    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6846d27-0053-4c6b-810a-1eab95f42638-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:06.874505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.874162    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6846d27-0053-4c6b-810a-1eab95f42638-metrics-certs\") pod \"maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:06.888329 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.888103    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdxsz\" (UniqueName: \"kubernetes.io/projected/f6846d27-0053-4c6b-810a-1eab95f42638-kube-api-access-wdxsz\") pod \"maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:06.945240 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:06.944750    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:07.259738 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:07.259660    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:07.306483 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:54:07.306442    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6846d27_0053_4c6b_810a_1eab95f42638.slice/crio-53dcf0a3a24418e7ddc498179ae7714c9216c5180b552a4b226bdb3b4509a074 WatchSource:0}: Error finding container 53dcf0a3a24418e7ddc498179ae7714c9216c5180b552a4b226bdb3b4509a074: Status 404 returned error can't find the container with id 53dcf0a3a24418e7ddc498179ae7714c9216c5180b552a4b226bdb3b4509a074
Oct 30 18:54:07.308367 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:07.308340    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"]
Oct 30 18:54:07.327243 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:07.325596    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 18:54:07.327243 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:07.325656    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74" podUID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 18:54:07.572559 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:07.572000    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng" event={"ID":"16453557-85f1-483b-8439-449950e8852d","Type":"ContainerStarted","Data":"11cb5d10cf09b626ba80f9424ef09e3e0171964d047eee5aea04e05c4ede5f54"}
Oct 30 18:54:07.572559 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:07.572212    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:07.577165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:07.575635    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4" event={"ID":"f6846d27-0053-4c6b-810a-1eab95f42638","Type":"ContainerStarted","Data":"53dcf0a3a24418e7ddc498179ae7714c9216c5180b552a4b226bdb3b4509a074"}
Oct 30 18:54:07.766180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:07.744988    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng" podStartSLOduration=3.292227764 podStartE2EDuration="3.744969354s" totalImagesPullingTime="452.74159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:07.681044271 +0000 UTC m=+1261.549462601" watchObservedRunningTime="2026-10-30 18:54:07.744969354 +0000 UTC m=+1261.613387685"
Oct 30 18:54:08.319878 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:08.319845    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:54:08.321215 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:08.321192    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:54:08.583384 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:08.583350    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4" event={"ID":"f6846d27-0053-4c6b-810a-1eab95f42638","Type":"ContainerStarted","Data":"c340bb6dedd27576066b73f3e219a3d18613215f50f6600531ab087959cc8eec"}
Oct 30 18:54:08.583775 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:08.583645    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:08.607772 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:08.607730    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4" podStartSLOduration=2.049621401 podStartE2EDuration="2.607717454s" totalImagesPullingTime="558.096053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:08.604044414 +0000 UTC m=+1262.472462949" watchObservedRunningTime="2026-10-30 18:54:08.607717454 +0000 UTC m=+1262.476135783"
Oct 30 18:54:09.453914 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:09.453886    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"
Oct 30 18:54:09.454090 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:09.453952    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"
Oct 30 18:54:09.869765 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:09.869732    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:54:11.489732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:11.489697    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"
Oct 30 18:54:13.589923 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:13.589887    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:54:14.595423 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:14.595392    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:54:15.260534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:15.260505    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:15.272899 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:15.272876    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:15.641362 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:15.641333    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:54:15.660265 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:15.660234    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"]
Oct 30 18:54:15.660486 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:15.660466    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t" podUID="de5bad8c-7124-450f-83aa-93b92384bd7a" containerName="istio-proxy" containerID="cri-o://61d19758fc7984cb53918841652d655f013401382968bc14b7f209d550747dd3" gracePeriod=30
Oct 30 18:54:15.664209 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:15.664183    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"]
Oct 30 18:54:15.664400 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:15.664377    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx" podUID="a363b854-6fb7-4d52-805f-c6aff76b100a" containerName="istio-proxy" containerID="cri-o://de6604fc739c10a50629510c44e3746cb4e2b53f0690ffb08bf51770ec90681c" gracePeriod=30
Oct 30 18:54:18.081621 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:18.081590    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"]
Oct 30 18:54:18.082421 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:18.082371    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv" podUID="ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" containerName="istio-proxy" containerID="cri-o://2603f53f28a8a57885876aba10a64ff5c8f3ba41451b81f3ee2bbdb92b405e4a" gracePeriod=30
Oct 30 18:54:18.087339 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:18.087298    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"]
Oct 30 18:54:18.087537 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:18.087495    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q" podUID="a5fb1131-af1f-4ce3-8822-c305e2767c89" containerName="istio-proxy" containerID="cri-o://c79682b4c5c91028a9edad9dea8569a5ba2276fb84b9e7572b3dda9e3a91889f" gracePeriod=30
Oct 30 18:54:20.783356 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:20.783319    4372 generic.go:356] "Generic (PLEG): container finished" podID="de5bad8c-7124-450f-83aa-93b92384bd7a" containerID="61d19758fc7984cb53918841652d655f013401382968bc14b7f209d550747dd3" exitCode=0
Oct 30 18:54:20.783705 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:20.783392    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t" event={"ID":"de5bad8c-7124-450f-83aa-93b92384bd7a","Type":"ContainerDied","Data":"61d19758fc7984cb53918841652d655f013401382968bc14b7f209d550747dd3"}
Oct 30 18:54:20.785556 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:20.785532    4372 generic.go:356] "Generic (PLEG): container finished" podID="a363b854-6fb7-4d52-805f-c6aff76b100a" containerID="de6604fc739c10a50629510c44e3746cb4e2b53f0690ffb08bf51770ec90681c" exitCode=0
Oct 30 18:54:20.785659 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:20.785612    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx" event={"ID":"a363b854-6fb7-4d52-805f-c6aff76b100a","Type":"ContainerDied","Data":"de6604fc739c10a50629510c44e3746cb4e2b53f0690ffb08bf51770ec90681c"}
Oct 30 18:54:20.953519 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:20.953442    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:54:21.032152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.030335    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:54:21.060702 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.060661    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-podinfo\") pod \"a363b854-6fb7-4d52-805f-c6aff76b100a\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") "
Oct 30 18:54:21.060896 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.060767    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-token\" (UniqueName: \"kubernetes.io/projected/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-token\") pod \"a363b854-6fb7-4d52-805f-c6aff76b100a\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") "
Oct 30 18:54:21.060896 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.060808    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a363b854-6fb7-4d52-805f-c6aff76b100a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a363b854-6fb7-4d52-805f-c6aff76b100a-istiod-ca-cert\") pod \"a363b854-6fb7-4d52-805f-c6aff76b100a\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") "
Oct 30 18:54:21.061040 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.060910    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-workload-certs\") pod \"a363b854-6fb7-4d52-805f-c6aff76b100a\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") "
Oct 30 18:54:21.061040 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.060942    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-envoy\") pod \"a363b854-6fb7-4d52-805f-c6aff76b100a\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") "
Oct 30 18:54:21.061174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.061090    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-workload-socket\") pod \"a363b854-6fb7-4d52-805f-c6aff76b100a\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") "
Oct 30 18:54:21.061236 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.061196    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-data\") pod \"a363b854-6fb7-4d52-805f-c6aff76b100a\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") "
Oct 30 18:54:21.062470 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.061305    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a363b854-6fb7-4d52-805f-c6aff76b100a-kube-api-access-fqrmr\" (UniqueName: \"kubernetes.io/projected/a363b854-6fb7-4d52-805f-c6aff76b100a-kube-api-access-fqrmr\") pod \"a363b854-6fb7-4d52-805f-c6aff76b100a\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") "
Oct 30 18:54:21.062470 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.061348    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-credential-socket\") pod \"a363b854-6fb7-4d52-805f-c6aff76b100a\" (UID: \"a363b854-6fb7-4d52-805f-c6aff76b100a\") "
Oct 30 18:54:21.062470 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.061909    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-workload-socket" pod "a363b854-6fb7-4d52-805f-c6aff76b100a" (UID: "a363b854-6fb7-4d52-805f-c6aff76b100a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:21.062470 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.062182    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.062470 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.062205    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-data" pod "a363b854-6fb7-4d52-805f-c6aff76b100a" (UID: "a363b854-6fb7-4d52-805f-c6aff76b100a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:21.064180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.062799    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-workload-certs" pod "a363b854-6fb7-4d52-805f-c6aff76b100a" (UID: "a363b854-6fb7-4d52-805f-c6aff76b100a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:21.065188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.064456    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a363b854-6fb7-4d52-805f-c6aff76b100a-istiod-ca-cert" pod "a363b854-6fb7-4d52-805f-c6aff76b100a" (UID: "a363b854-6fb7-4d52-805f-c6aff76b100a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:21.068985 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.068957    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-podinfo" pod "a363b854-6fb7-4d52-805f-c6aff76b100a" (UID: "a363b854-6fb7-4d52-805f-c6aff76b100a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:54:21.073114 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.073087    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a363b854-6fb7-4d52-805f-c6aff76b100a-kube-api-access-fqrmr" pod "a363b854-6fb7-4d52-805f-c6aff76b100a" (UID: "a363b854-6fb7-4d52-805f-c6aff76b100a"). InnerVolumeSpecName "kube-api-access-fqrmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:21.078404 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.078377    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-credential-socket" pod "a363b854-6fb7-4d52-805f-c6aff76b100a" (UID: "a363b854-6fb7-4d52-805f-c6aff76b100a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:21.083624 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.083496    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-envoy" pod "a363b854-6fb7-4d52-805f-c6aff76b100a" (UID: "a363b854-6fb7-4d52-805f-c6aff76b100a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:21.086026 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.085995    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-token" pod "a363b854-6fb7-4d52-805f-c6aff76b100a" (UID: "a363b854-6fb7-4d52-805f-c6aff76b100a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:21.163457 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.163429    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-token\" (UniqueName: \"kubernetes.io/projected/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-token\") pod \"de5bad8c-7124-450f-83aa-93b92384bd7a\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") "
Oct 30 18:54:21.163571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.163491    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-workload-socket\") pod \"de5bad8c-7124-450f-83aa-93b92384bd7a\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") "
Oct 30 18:54:21.163571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.163524    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-envoy\") pod \"de5bad8c-7124-450f-83aa-93b92384bd7a\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") "
Oct 30 18:54:21.163674 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.163661    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5bad8c-7124-450f-83aa-93b92384bd7a-kube-api-access-kknmg\" (UniqueName: \"kubernetes.io/projected/de5bad8c-7124-450f-83aa-93b92384bd7a-kube-api-access-kknmg\") pod \"de5bad8c-7124-450f-83aa-93b92384bd7a\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") "
Oct 30 18:54:21.163717 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.163698    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-podinfo\") pod \"de5bad8c-7124-450f-83aa-93b92384bd7a\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") "
Oct 30 18:54:21.163771 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.163737    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-workload-certs\") pod \"de5bad8c-7124-450f-83aa-93b92384bd7a\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") "
Oct 30 18:54:21.163825 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.163768    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-workload-socket" pod "de5bad8c-7124-450f-83aa-93b92384bd7a" (UID: "de5bad8c-7124-450f-83aa-93b92384bd7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:21.163886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.163855    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-credential-socket\") pod \"de5bad8c-7124-450f-83aa-93b92384bd7a\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") "
Oct 30 18:54:21.163980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.163961    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-data\") pod \"de5bad8c-7124-450f-83aa-93b92384bd7a\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") "
Oct 30 18:54:21.164048 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164001    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de5bad8c-7124-450f-83aa-93b92384bd7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5bad8c-7124-450f-83aa-93b92384bd7a-istiod-ca-cert\") pod \"de5bad8c-7124-450f-83aa-93b92384bd7a\" (UID: \"de5bad8c-7124-450f-83aa-93b92384bd7a\") "
Oct 30 18:54:21.164184 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164154    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-credential-socket" pod "de5bad8c-7124-450f-83aa-93b92384bd7a" (UID: "de5bad8c-7124-450f-83aa-93b92384bd7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:21.164313 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164292    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-data" pod "de5bad8c-7124-450f-83aa-93b92384bd7a" (UID: "de5bad8c-7124-450f-83aa-93b92384bd7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:21.164965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164523    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-workload-certs" pod "de5bad8c-7124-450f-83aa-93b92384bd7a" (UID: "de5bad8c-7124-450f-83aa-93b92384bd7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:21.164965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164691    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5bad8c-7124-450f-83aa-93b92384bd7a-istiod-ca-cert" pod "de5bad8c-7124-450f-83aa-93b92384bd7a" (UID: "de5bad8c-7124-450f-83aa-93b92384bd7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:21.164965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164840    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.164965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164852    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.164965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164862    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.164965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164871    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqrmr\" (UniqueName: \"kubernetes.io/projected/a363b854-6fb7-4d52-805f-c6aff76b100a-kube-api-access-fqrmr\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.164965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164881    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.164965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164889    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.164965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164898    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.164965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164906    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.164965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164914    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5bad8c-7124-450f-83aa-93b92384bd7a-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.164965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164923    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a363b854-6fb7-4d52-805f-c6aff76b100a-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.164965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164931    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a363b854-6fb7-4d52-805f-c6aff76b100a-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.164965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164940    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.164965 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.164949    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a363b854-6fb7-4d52-805f-c6aff76b100a-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.165829 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.165806    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-envoy" pod "de5bad8c-7124-450f-83aa-93b92384bd7a" (UID: "de5bad8c-7124-450f-83aa-93b92384bd7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:21.165994 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.165969    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-podinfo" pod "de5bad8c-7124-450f-83aa-93b92384bd7a" (UID: "de5bad8c-7124-450f-83aa-93b92384bd7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:54:21.166224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.166206    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-token" pod "de5bad8c-7124-450f-83aa-93b92384bd7a" (UID: "de5bad8c-7124-450f-83aa-93b92384bd7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:21.166467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.166450    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5bad8c-7124-450f-83aa-93b92384bd7a-kube-api-access-kknmg" pod "de5bad8c-7124-450f-83aa-93b92384bd7a" (UID: "de5bad8c-7124-450f-83aa-93b92384bd7a"). InnerVolumeSpecName "kube-api-access-kknmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:21.282856 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.276232    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.282856 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.276268    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.282856 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.276283    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kknmg\" (UniqueName: \"kubernetes.io/projected/de5bad8c-7124-450f-83aa-93b92384bd7a-kube-api-access-kknmg\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.282856 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.276301    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5bad8c-7124-450f-83aa-93b92384bd7a-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.793685 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.793662    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"
Oct 30 18:54:21.794067 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.793695    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx" event={"ID":"a363b854-6fb7-4d52-805f-c6aff76b100a","Type":"ContainerDied","Data":"99656f31db899723457e2cc0cf9c956b0bbf767dca493a772e4ce8eae6747586"}
Oct 30 18:54:21.794067 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.793755    4372 scope.go:122] "RemoveContainer" containerID="de6604fc739c10a50629510c44e3746cb4e2b53f0690ffb08bf51770ec90681c"
Oct 30 18:54:21.795508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.795487    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t" event={"ID":"de5bad8c-7124-450f-83aa-93b92384bd7a","Type":"ContainerDied","Data":"9d0acde1720c8efd8ba27c955cfb85f1ff4b9324689e81ec86baa04e28f8df3c"}
Oct 30 18:54:21.795508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.795492    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"
Oct 30 18:54:21.811440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.804425    4372 scope.go:122] "RemoveContainer" containerID="61d19758fc7984cb53918841652d655f013401382968bc14b7f209d550747dd3"
Oct 30 18:54:21.849203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.849176    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"]
Oct 30 18:54:21.855305 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.855282    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-brs8t"]
Oct 30 18:54:21.870873 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.870842    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"]
Oct 30 18:54:21.876448 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:21.876428    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-sfgbx"]
Oct 30 18:54:22.267453 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:22.267426    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a363b854-6fb7-4d52-805f-c6aff76b100a" path="/var/lib/kubelet/pods/a363b854-6fb7-4d52-805f-c6aff76b100a/volumes"
Oct 30 18:54:22.267878 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:22.267864    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5bad8c-7124-450f-83aa-93b92384bd7a" path="/var/lib/kubelet/pods/de5bad8c-7124-450f-83aa-93b92384bd7a/volumes"
Oct 30 18:54:23.433008 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.432979    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:54:23.459410 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.459390    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:54:23.515435 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.515396    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-data\") pod \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") "
Oct 30 18:54:23.515588 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.515462    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-kube-api-access-lnmfd\" (UniqueName: \"kubernetes.io/projected/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-kube-api-access-lnmfd\") pod \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") "
Oct 30 18:54:23.515588 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.515495    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-podinfo\") pod \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") "
Oct 30 18:54:23.515588 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.515569    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-credential-socket\") pod \"a5fb1131-af1f-4ce3-8822-c305e2767c89\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") "
Oct 30 18:54:23.515792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.515598    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-workload-socket\") pod \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") "
Oct 30 18:54:23.515792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.515637    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-podinfo\") pod \"a5fb1131-af1f-4ce3-8822-c305e2767c89\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") "
Oct 30 18:54:23.515792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.515677    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-workload-certs\") pod \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") "
Oct 30 18:54:23.515792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.515733    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-data\") pod \"a5fb1131-af1f-4ce3-8822-c305e2767c89\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") "
Oct 30 18:54:23.516004 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.515833    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-workload-socket\") pod \"a5fb1131-af1f-4ce3-8822-c305e2767c89\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") "
Oct 30 18:54:23.516004 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.515898    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-envoy\") pod \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") "
Oct 30 18:54:23.516004 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.515933    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istiod-ca-cert\") pod \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") "
Oct 30 18:54:23.516209 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.516021    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-envoy\") pod \"a5fb1131-af1f-4ce3-8822-c305e2767c89\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") "
Oct 30 18:54:23.516209 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.516053    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5fb1131-af1f-4ce3-8822-c305e2767c89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5fb1131-af1f-4ce3-8822-c305e2767c89-istiod-ca-cert\") pod \"a5fb1131-af1f-4ce3-8822-c305e2767c89\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") "
Oct 30 18:54:23.516209 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.516087    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-credential-socket\") pod \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") "
Oct 30 18:54:23.516379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.516209    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fb1131-af1f-4ce3-8822-c305e2767c89-kube-api-access-wh9vs\" (UniqueName: \"kubernetes.io/projected/a5fb1131-af1f-4ce3-8822-c305e2767c89-kube-api-access-wh9vs\") pod \"a5fb1131-af1f-4ce3-8822-c305e2767c89\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") "
Oct 30 18:54:23.516379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.516259    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-token\" (UniqueName: \"kubernetes.io/projected/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-token\") pod \"a5fb1131-af1f-4ce3-8822-c305e2767c89\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") "
Oct 30 18:54:23.516379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.516347    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-workload-certs\") pod \"a5fb1131-af1f-4ce3-8822-c305e2767c89\" (UID: \"a5fb1131-af1f-4ce3-8822-c305e2767c89\") "
Oct 30 18:54:23.516557 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.516385    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-workload-socket" pod "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" (UID: "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:23.516557 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.516419    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-token\" (UniqueName: \"kubernetes.io/projected/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-token\") pod \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\" (UID: \"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284\") "
Oct 30 18:54:23.516557 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.516502    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-data" pod "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" (UID: "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:23.516760 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.516570    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-credential-socket" pod "a5fb1131-af1f-4ce3-8822-c305e2767c89" (UID: "a5fb1131-af1f-4ce3-8822-c305e2767c89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:23.516760 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.516643    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-data" pod "a5fb1131-af1f-4ce3-8822-c305e2767c89" (UID: "a5fb1131-af1f-4ce3-8822-c305e2767c89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:23.516904 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.516881    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-credential-socket" pod "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" (UID: "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:23.516971 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.516919    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-workload-certs" pod "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" (UID: "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:23.517025 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.516973    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-workload-socket" pod "a5fb1131-af1f-4ce3-8822-c305e2767c89" (UID: "a5fb1131-af1f-4ce3-8822-c305e2767c89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:23.517615 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.517225    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-workload-certs" pod "a5fb1131-af1f-4ce3-8822-c305e2767c89" (UID: "a5fb1131-af1f-4ce3-8822-c305e2767c89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:23.517615 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.517260    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istiod-ca-cert" pod "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" (UID: "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:23.520111 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.520070    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5fb1131-af1f-4ce3-8822-c305e2767c89-istiod-ca-cert" pod "a5fb1131-af1f-4ce3-8822-c305e2767c89" (UID: "a5fb1131-af1f-4ce3-8822-c305e2767c89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:23.520393 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.520317    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.520393 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.520347    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.520393 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.520362    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.520798 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.520743    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.520798 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.520771    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.521239 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.520999    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.521239 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.521020    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.521239 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.521034    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.521239 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.521048    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.522606 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.522519    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-token" pod "a5fb1131-af1f-4ce3-8822-c305e2767c89" (UID: "a5fb1131-af1f-4ce3-8822-c305e2767c89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:23.522721 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.522663    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-podinfo" pod "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" (UID: "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:54:23.524390 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.524355    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-envoy" pod "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" (UID: "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:23.524851 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.524817    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-podinfo" pod "a5fb1131-af1f-4ce3-8822-c305e2767c89" (UID: "a5fb1131-af1f-4ce3-8822-c305e2767c89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:54:23.525075 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.525048    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-kube-api-access-lnmfd" pod "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" (UID: "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284"). InnerVolumeSpecName "kube-api-access-lnmfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:23.525592 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.525565    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-token" pod "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" (UID: "ba48ce60-30f0-4bb7-8d76-bbb4a32ae284"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:23.526307 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.525833    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-envoy" pod "a5fb1131-af1f-4ce3-8822-c305e2767c89" (UID: "a5fb1131-af1f-4ce3-8822-c305e2767c89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:23.526307 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.526158    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fb1131-af1f-4ce3-8822-c305e2767c89-kube-api-access-wh9vs" pod "a5fb1131-af1f-4ce3-8822-c305e2767c89" (UID: "a5fb1131-af1f-4ce3-8822-c305e2767c89"). InnerVolumeSpecName "kube-api-access-wh9vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:23.622887 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.622857    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.622887 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.622885    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnmfd\" (UniqueName: \"kubernetes.io/projected/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-kube-api-access-lnmfd\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.623023 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.622895    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.623023 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.622905    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.623023 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.622913    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.623023 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.622923    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.623023 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.622931    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5fb1131-af1f-4ce3-8822-c305e2767c89-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.623023 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.622940    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh9vs\" (UniqueName: \"kubernetes.io/projected/a5fb1131-af1f-4ce3-8822-c305e2767c89-kube-api-access-wh9vs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.623023 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.622949    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5fb1131-af1f-4ce3-8822-c305e2767c89-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.865151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.856321    4372 generic.go:356] "Generic (PLEG): container finished" podID="ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" containerID="2603f53f28a8a57885876aba10a64ff5c8f3ba41451b81f3ee2bbdb92b405e4a" exitCode=0
Oct 30 18:54:23.865151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.856463    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv" event={"ID":"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284","Type":"ContainerDied","Data":"2603f53f28a8a57885876aba10a64ff5c8f3ba41451b81f3ee2bbdb92b405e4a"}
Oct 30 18:54:23.865151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.856515    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv" event={"ID":"ba48ce60-30f0-4bb7-8d76-bbb4a32ae284","Type":"ContainerDied","Data":"3a4aa3ef8db54a929abdf35df501d977aa85a2dc34477305b63a6cdf04da73f0"}
Oct 30 18:54:23.865151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.856542    4372 scope.go:122] "RemoveContainer" containerID="2603f53f28a8a57885876aba10a64ff5c8f3ba41451b81f3ee2bbdb92b405e4a"
Oct 30 18:54:23.865151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.856692    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"
Oct 30 18:54:23.873027 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.872996    4372 generic.go:356] "Generic (PLEG): container finished" podID="a5fb1131-af1f-4ce3-8822-c305e2767c89" containerID="c79682b4c5c91028a9edad9dea8569a5ba2276fb84b9e7572b3dda9e3a91889f" exitCode=0
Oct 30 18:54:23.873405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.873362    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q" event={"ID":"a5fb1131-af1f-4ce3-8822-c305e2767c89","Type":"ContainerDied","Data":"c79682b4c5c91028a9edad9dea8569a5ba2276fb84b9e7572b3dda9e3a91889f"}
Oct 30 18:54:23.873600 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.873575    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q" event={"ID":"a5fb1131-af1f-4ce3-8822-c305e2767c89","Type":"ContainerDied","Data":"41c1fd5553dab09e0dc9392663a6a1e9361bc8b7f3fda19c2014a73dff932993"}
Oct 30 18:54:23.873800 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.873784    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"
Oct 30 18:54:23.965964 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.965926    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"]
Oct 30 18:54:23.966336 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.966305    4372 scope.go:122] "RemoveContainer" containerID="2603f53f28a8a57885876aba10a64ff5c8f3ba41451b81f3ee2bbdb92b405e4a"
Oct 30 18:54:23.966835 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:54:23.966809    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2603f53f28a8a57885876aba10a64ff5c8f3ba41451b81f3ee2bbdb92b405e4a\": container with ID starting with 2603f53f28a8a57885876aba10a64ff5c8f3ba41451b81f3ee2bbdb92b405e4a not found: ID does not exist" containerID="2603f53f28a8a57885876aba10a64ff5c8f3ba41451b81f3ee2bbdb92b405e4a"
Oct 30 18:54:23.966956 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.966858    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2603f53f28a8a57885876aba10a64ff5c8f3ba41451b81f3ee2bbdb92b405e4a"} err="failed to get container status \"2603f53f28a8a57885876aba10a64ff5c8f3ba41451b81f3ee2bbdb92b405e4a\": rpc error: code = NotFound desc = could not find container \"2603f53f28a8a57885876aba10a64ff5c8f3ba41451b81f3ee2bbdb92b405e4a\": container with ID starting with 2603f53f28a8a57885876aba10a64ff5c8f3ba41451b81f3ee2bbdb92b405e4a not found: ID does not exist"
Oct 30 18:54:23.966956 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.966882    4372 scope.go:122] "RemoveContainer" containerID="c79682b4c5c91028a9edad9dea8569a5ba2276fb84b9e7572b3dda9e3a91889f"
Oct 30 18:54:23.972632 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.972612    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bqs6q"]
Oct 30 18:54:23.985065 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.985037    4372 scope.go:122] "RemoveContainer" containerID="c79682b4c5c91028a9edad9dea8569a5ba2276fb84b9e7572b3dda9e3a91889f"
Oct 30 18:54:23.985538 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:54:23.985514    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79682b4c5c91028a9edad9dea8569a5ba2276fb84b9e7572b3dda9e3a91889f\": container with ID starting with c79682b4c5c91028a9edad9dea8569a5ba2276fb84b9e7572b3dda9e3a91889f not found: ID does not exist" containerID="c79682b4c5c91028a9edad9dea8569a5ba2276fb84b9e7572b3dda9e3a91889f"
Oct 30 18:54:23.985644 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.985549    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79682b4c5c91028a9edad9dea8569a5ba2276fb84b9e7572b3dda9e3a91889f"} err="failed to get container status \"c79682b4c5c91028a9edad9dea8569a5ba2276fb84b9e7572b3dda9e3a91889f\": rpc error: code = NotFound desc = could not find container \"c79682b4c5c91028a9edad9dea8569a5ba2276fb84b9e7572b3dda9e3a91889f\": container with ID starting with c79682b4c5c91028a9edad9dea8569a5ba2276fb84b9e7572b3dda9e3a91889f not found: ID does not exist"
Oct 30 18:54:24.000028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:23.999940    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"]
Oct 30 18:54:24.016419 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.015487    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-lcbtv"]
Oct 30 18:54:24.266802 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.266776    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5fb1131-af1f-4ce3-8822-c305e2767c89" path="/var/lib/kubelet/pods/a5fb1131-af1f-4ce3-8822-c305e2767c89/volumes"
Oct 30 18:54:24.267322 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.267308    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" path="/var/lib/kubelet/pods/ba48ce60-30f0-4bb7-8d76-bbb4a32ae284/volumes"
Oct 30 18:54:24.872804 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.872767    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"]
Oct 30 18:54:24.873424 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.873408    4372 cpu_manager.go:414] "containerMap: removing container" podUID="a363b854-6fb7-4d52-805f-c6aff76b100a" containerName="istio-proxy"
Oct 30 18:54:24.873483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.873426    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a363b854-6fb7-4d52-805f-c6aff76b100a" containerName="istio-proxy"
Oct 30 18:54:24.873483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.873450    4372 cpu_manager.go:414] "containerMap: removing container" podUID="a5fb1131-af1f-4ce3-8822-c305e2767c89" containerName="istio-proxy"
Oct 30 18:54:24.873483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.873455    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fb1131-af1f-4ce3-8822-c305e2767c89" containerName="istio-proxy"
Oct 30 18:54:24.873483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.873473    4372 cpu_manager.go:414] "containerMap: removing container" podUID="de5bad8c-7124-450f-83aa-93b92384bd7a" containerName="istio-proxy"
Oct 30 18:54:24.873483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.873478    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5bad8c-7124-450f-83aa-93b92384bd7a" containerName="istio-proxy"
Oct 30 18:54:24.873638 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.873490    4372 cpu_manager.go:414] "containerMap: removing container" podUID="ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" containerName="istio-proxy"
Oct 30 18:54:24.873638 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.873495    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" containerName="istio-proxy"
Oct 30 18:54:24.873638 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.873576    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fb1131-af1f-4ce3-8822-c305e2767c89" containerName="istio-proxy"
Oct 30 18:54:24.873638 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.873589    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5bad8c-7124-450f-83aa-93b92384bd7a" containerName="istio-proxy"
Oct 30 18:54:24.873638 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.873595    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba48ce60-30f0-4bb7-8d76-bbb4a32ae284" containerName="istio-proxy"
Oct 30 18:54:24.873638 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.873603    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="a363b854-6fb7-4d52-805f-c6aff76b100a" containerName="istio-proxy"
Oct 30 18:54:24.878705 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.878684    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:24.911170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.910620    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-9d42ff-openshift-default-dockercfg-qnmrm\""
Oct 30 18:54:24.914377 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.914354    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"]
Oct 30 18:54:24.979889 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.979845    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-data\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:24.979889 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.979888    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-workload-socket\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:24.980171 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.979929    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-podinfo\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:24.980171 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.979963    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istiod-ca-cert\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:24.980171 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.980022    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-workload-certs\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:24.980171 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.980112    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-token\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:24.992584 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.986320    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-envoy\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:24.992584 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.986400    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwk9z\" (UniqueName: \"kubernetes.io/projected/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-kube-api-access-xwk9z\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:24.992584 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:24.986574    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-credential-socket\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.092150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.089070    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwk9z\" (UniqueName: \"kubernetes.io/projected/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-kube-api-access-xwk9z\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.092150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.089295    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-credential-socket\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.092150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.089425    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-data\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.092150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.089464    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-workload-socket\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.092150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.089527    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-podinfo\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.092150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.089591    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istiod-ca-cert\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.092150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.089675    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-workload-certs\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.092150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.089777    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-token\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.092150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.089804    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-envoy\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.092150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.090723    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-workload-socket\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.092150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.091428    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-credential-socket\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.092150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.091666    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-data\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.092985 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.092370    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istiod-ca-cert\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.113146 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.103611    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-podinfo\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.113146 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.103875    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-workload-certs\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.126846 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.126196    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-envoy\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.147149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.145933    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwk9z\" (UniqueName: \"kubernetes.io/projected/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-kube-api-access-xwk9z\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.147304 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.147191    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-token\") pod \"e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") " pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.192676 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.192639    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:25.537789 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.537715    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"]
Oct 30 18:54:25.557397 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:54:25.557368    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2045c19b_ea71_4ba2_9b6f_fdfe9280e2f4.slice/crio-6a0dec251b5363fd9708d1e37278749b02b10605e6772fdab5d33aa49a1de5c9 WatchSource:0}: Error finding container 6a0dec251b5363fd9708d1e37278749b02b10605e6772fdab5d33aa49a1de5c9: Status 404 returned error can't find the container with id 6a0dec251b5363fd9708d1e37278749b02b10605e6772fdab5d33aa49a1de5c9
Oct 30 18:54:25.560027 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.559987    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:54:25.560153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.560073    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:54:25.560153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.560142    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:54:25.891505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.891467    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh" event={"ID":"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4","Type":"ContainerStarted","Data":"531c3282d0ffad66fa9b62638c247e4d7124c4b1ecb08c3438e4c5936fabef36"}
Oct 30 18:54:25.891953 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.891531    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh" event={"ID":"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4","Type":"ContainerStarted","Data":"6a0dec251b5363fd9708d1e37278749b02b10605e6772fdab5d33aa49a1de5c9"}
Oct 30 18:54:25.919799 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:25.919754    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh" podStartSLOduration=1.919741685 podStartE2EDuration="1.919741685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:25.919574521 +0000 UTC m=+1279.787992851" watchObservedRunningTime="2026-10-30 18:54:25.919741685 +0000 UTC m=+1279.788160015"
Oct 30 18:54:26.193311 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:26.193232    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:26.198913 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:26.198890    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:26.898220 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:26.898189    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:26.899117 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:26.899100    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:54:27.696338 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:54:27.696300    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:54:28.056472 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:28.056397    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"]
Oct 30 18:54:28.056874 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:28.056595    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9" podUID="46273550-7eb9-47ba-8216-3d586b219ef3" containerName="istio-proxy" containerID="cri-o://eda093a54590c6fe28ede98d7a26b376e130648381ad9bca5c82426a9af594e7" gracePeriod=30
Oct 30 18:54:28.059914 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:28.059892    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"]
Oct 30 18:54:28.060095 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:28.060073    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7" podUID="09619584-4053-472a-9723-3dce678c9f4c" containerName="istio-proxy" containerID="cri-o://e72e71e827125b14124aa108ec9c48dfed1e0869f0d39bfd2171f7681908fcca" gracePeriod=30
Oct 30 18:54:29.194724 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.194690    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"]
Oct 30 18:54:29.200181 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.200161    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"
Oct 30 18:54:29.213252 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.213227    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"]
Oct 30 18:54:29.213351 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.213323    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-59f140\""
Oct 30 18:54:29.217121 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.217106    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"
Oct 30 18:54:29.227380 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.227357    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"]
Oct 30 18:54:29.245208 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.245184    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"]
Oct 30 18:54:29.256355 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.256335    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-59f140-dockercfg-wdmbl\""
Oct 30 18:54:29.285873 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.285850    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z8qn\" (UniqueName: \"kubernetes.io/projected/a513a47e-5603-4b6c-a1e1-48bcd0c85a32-kube-api-access-9z8qn\") pod \"payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl\" (UID: \"a513a47e-5603-4b6c-a1e1-48bcd0c85a32\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"
Oct 30 18:54:29.285979 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.285919    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a0503b0-52db-42f1-9904-fa6395aeb865-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m\" (UID: \"8a0503b0-52db-42f1-9904-fa6395aeb865\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"
Oct 30 18:54:29.286136 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.286107    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a513a47e-5603-4b6c-a1e1-48bcd0c85a32-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl\" (UID: \"a513a47e-5603-4b6c-a1e1-48bcd0c85a32\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"
Oct 30 18:54:29.286178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.286170    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rls4d\" (UniqueName: \"kubernetes.io/projected/8a0503b0-52db-42f1-9904-fa6395aeb865-kube-api-access-rls4d\") pod \"payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m\" (UID: \"8a0503b0-52db-42f1-9904-fa6395aeb865\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"
Oct 30 18:54:29.396151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.395144    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z8qn\" (UniqueName: \"kubernetes.io/projected/a513a47e-5603-4b6c-a1e1-48bcd0c85a32-kube-api-access-9z8qn\") pod \"payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl\" (UID: \"a513a47e-5603-4b6c-a1e1-48bcd0c85a32\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"
Oct 30 18:54:29.396151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.395251    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a0503b0-52db-42f1-9904-fa6395aeb865-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m\" (UID: \"8a0503b0-52db-42f1-9904-fa6395aeb865\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"
Oct 30 18:54:29.396151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.395448    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a513a47e-5603-4b6c-a1e1-48bcd0c85a32-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl\" (UID: \"a513a47e-5603-4b6c-a1e1-48bcd0c85a32\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"
Oct 30 18:54:29.396151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.395509    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rls4d\" (UniqueName: \"kubernetes.io/projected/8a0503b0-52db-42f1-9904-fa6395aeb865-kube-api-access-rls4d\") pod \"payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m\" (UID: \"8a0503b0-52db-42f1-9904-fa6395aeb865\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"
Oct 30 18:54:29.416152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.397077    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a0503b0-52db-42f1-9904-fa6395aeb865-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m\" (UID: \"8a0503b0-52db-42f1-9904-fa6395aeb865\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"
Oct 30 18:54:29.416152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.402414    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a513a47e-5603-4b6c-a1e1-48bcd0c85a32-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl\" (UID: \"a513a47e-5603-4b6c-a1e1-48bcd0c85a32\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"
Oct 30 18:54:29.446869 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.446778    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rls4d\" (UniqueName: \"kubernetes.io/projected/8a0503b0-52db-42f1-9904-fa6395aeb865-kube-api-access-rls4d\") pod \"payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m\" (UID: \"8a0503b0-52db-42f1-9904-fa6395aeb865\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"
Oct 30 18:54:29.464267 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.464237    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z8qn\" (UniqueName: \"kubernetes.io/projected/a513a47e-5603-4b6c-a1e1-48bcd0c85a32-kube-api-access-9z8qn\") pod \"payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl\" (UID: \"a513a47e-5603-4b6c-a1e1-48bcd0c85a32\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"
Oct 30 18:54:29.515247 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.514507    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"
Oct 30 18:54:29.544555 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.541785    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"
Oct 30 18:54:29.898261 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:54:29.898232    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0503b0_52db_42f1_9904_fa6395aeb865.slice/crio-f03461a87c5041e30de69af8a2756acfcb308b72218bbae115e60940ef82a0ef WatchSource:0}: Error finding container f03461a87c5041e30de69af8a2756acfcb308b72218bbae115e60940ef82a0ef: Status 404 returned error can't find the container with id f03461a87c5041e30de69af8a2756acfcb308b72218bbae115e60940ef82a0ef
Oct 30 18:54:29.900223 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.900094    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"]
Oct 30 18:54:29.900975 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:54:29.900956    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda513a47e_5603_4b6c_a1e1_48bcd0c85a32.slice/crio-8aafc8714cc0f922bdfd798d7973aebc3027a238f56f4f4d3904f3076ace6cc0 WatchSource:0}: Error finding container 8aafc8714cc0f922bdfd798d7973aebc3027a238f56f4f4d3904f3076ace6cc0: Status 404 returned error can't find the container with id 8aafc8714cc0f922bdfd798d7973aebc3027a238f56f4f4d3904f3076ace6cc0
Oct 30 18:54:29.904637 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.904617    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"]
Oct 30 18:54:29.923089 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.922968    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m" event={"ID":"8a0503b0-52db-42f1-9904-fa6395aeb865","Type":"ContainerStarted","Data":"f03461a87c5041e30de69af8a2756acfcb308b72218bbae115e60940ef82a0ef"}
Oct 30 18:54:29.925064 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:29.925031    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl" event={"ID":"a513a47e-5603-4b6c-a1e1-48bcd0c85a32","Type":"ContainerStarted","Data":"8aafc8714cc0f922bdfd798d7973aebc3027a238f56f4f4d3904f3076ace6cc0"}
Oct 30 18:54:30.934075 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:30.934034    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m" event={"ID":"8a0503b0-52db-42f1-9904-fa6395aeb865","Type":"ContainerStarted","Data":"a7bd6ebc484f7735b416178c341f9bbbd94716f1c2ea62ca910d964f9b8f1f42"}
Oct 30 18:54:30.934570 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:30.934114    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"
Oct 30 18:54:30.935736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:30.935712    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl" event={"ID":"a513a47e-5603-4b6c-a1e1-48bcd0c85a32","Type":"ContainerStarted","Data":"9b9fa464ea87f95cffe063e91b0bf6102489a7632ac45abf6e120016df681461"}
Oct 30 18:54:30.935831 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:30.935763    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"
Oct 30 18:54:30.958133 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:30.958085    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m" podStartSLOduration=1.958072428 podStartE2EDuration="1.958072428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:30.957044647 +0000 UTC m=+1284.825462971" watchObservedRunningTime="2026-10-30 18:54:30.958072428 +0000 UTC m=+1284.826490758"
Oct 30 18:54:30.983155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:30.983080    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl" podStartSLOduration=1.983060568 podStartE2EDuration="1.983060568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:30.98106735 +0000 UTC m=+1284.849485680" watchObservedRunningTime="2026-10-30 18:54:30.983060568 +0000 UTC m=+1284.851478901"
Oct 30 18:54:33.355161 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.355113    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:54:33.379095 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.379067    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46273550-7eb9-47ba-8216-3d586b219ef3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46273550-7eb9-47ba-8216-3d586b219ef3-istio-podinfo\") pod \"46273550-7eb9-47ba-8216-3d586b219ef3\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") "
Oct 30 18:54:33.379230 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.379115    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-workload-socket\") pod \"46273550-7eb9-47ba-8216-3d586b219ef3\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") "
Oct 30 18:54:33.379230 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.379170    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-workload-certs\") pod \"46273550-7eb9-47ba-8216-3d586b219ef3\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") "
Oct 30 18:54:33.379230 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.379199    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-credential-socket\") pod \"46273550-7eb9-47ba-8216-3d586b219ef3\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") "
Oct 30 18:54:33.379400 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.379262    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46273550-7eb9-47ba-8216-3d586b219ef3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46273550-7eb9-47ba-8216-3d586b219ef3-istiod-ca-cert\") pod \"46273550-7eb9-47ba-8216-3d586b219ef3\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") "
Oct 30 18:54:33.379400 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.379326    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46273550-7eb9-47ba-8216-3d586b219ef3-kube-api-access-9m5sj\" (UniqueName: \"kubernetes.io/projected/46273550-7eb9-47ba-8216-3d586b219ef3-kube-api-access-9m5sj\") pod \"46273550-7eb9-47ba-8216-3d586b219ef3\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") "
Oct 30 18:54:33.379400 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.379378    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-istio-envoy\") pod \"46273550-7eb9-47ba-8216-3d586b219ef3\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") "
Oct 30 18:54:33.379541 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.379449    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-workload-certs" pod "46273550-7eb9-47ba-8216-3d586b219ef3" (UID: "46273550-7eb9-47ba-8216-3d586b219ef3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:33.379541 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.379460    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-workload-socket" pod "46273550-7eb9-47ba-8216-3d586b219ef3" (UID: "46273550-7eb9-47ba-8216-3d586b219ef3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:33.379541 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.379483    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-credential-socket" pod "46273550-7eb9-47ba-8216-3d586b219ef3" (UID: "46273550-7eb9-47ba-8216-3d586b219ef3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:33.379690 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.379562    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46273550-7eb9-47ba-8216-3d586b219ef3-istio-token\" (UniqueName: \"kubernetes.io/projected/46273550-7eb9-47ba-8216-3d586b219ef3-istio-token\") pod \"46273550-7eb9-47ba-8216-3d586b219ef3\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") "
Oct 30 18:54:33.379690 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.379652    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-istio-data\") pod \"46273550-7eb9-47ba-8216-3d586b219ef3\" (UID: \"46273550-7eb9-47ba-8216-3d586b219ef3\") "
Oct 30 18:54:33.379793 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.379718    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46273550-7eb9-47ba-8216-3d586b219ef3-istiod-ca-cert" pod "46273550-7eb9-47ba-8216-3d586b219ef3" (UID: "46273550-7eb9-47ba-8216-3d586b219ef3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:33.380559 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.380034    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-istio-data" pod "46273550-7eb9-47ba-8216-3d586b219ef3" (UID: "46273550-7eb9-47ba-8216-3d586b219ef3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:33.380559 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.380459    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.380559 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.380483    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.380559 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.380500    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.380559 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.380514    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.380559 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.380530    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46273550-7eb9-47ba-8216-3d586b219ef3-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.381838 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.381796    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46273550-7eb9-47ba-8216-3d586b219ef3-istio-podinfo" pod "46273550-7eb9-47ba-8216-3d586b219ef3" (UID: "46273550-7eb9-47ba-8216-3d586b219ef3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:54:33.383116 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.383089    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-istio-envoy" pod "46273550-7eb9-47ba-8216-3d586b219ef3" (UID: "46273550-7eb9-47ba-8216-3d586b219ef3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:33.383238 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.383166    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46273550-7eb9-47ba-8216-3d586b219ef3-istio-token" pod "46273550-7eb9-47ba-8216-3d586b219ef3" (UID: "46273550-7eb9-47ba-8216-3d586b219ef3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:33.388471 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.388449    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46273550-7eb9-47ba-8216-3d586b219ef3-kube-api-access-9m5sj" pod "46273550-7eb9-47ba-8216-3d586b219ef3" (UID: "46273550-7eb9-47ba-8216-3d586b219ef3"). InnerVolumeSpecName "kube-api-access-9m5sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:33.400938 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.400911    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:54:33.481926 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.481892    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09619584-4053-472a-9723-3dce678c9f4c-kube-api-access-jncbz\" (UniqueName: \"kubernetes.io/projected/09619584-4053-472a-9723-3dce678c9f4c-kube-api-access-jncbz\") pod \"09619584-4053-472a-9723-3dce678c9f4c\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") "
Oct 30 18:54:33.482096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.481947    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-istio-data\") pod \"09619584-4053-472a-9723-3dce678c9f4c\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") "
Oct 30 18:54:33.482096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.482030    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09619584-4053-472a-9723-3dce678c9f4c-istio-token\" (UniqueName: \"kubernetes.io/projected/09619584-4053-472a-9723-3dce678c9f4c-istio-token\") pod \"09619584-4053-472a-9723-3dce678c9f4c\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") "
Oct 30 18:54:33.482402 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.482153    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-workload-socket\") pod \"09619584-4053-472a-9723-3dce678c9f4c\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") "
Oct 30 18:54:33.482402 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.482193    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-credential-socket\") pod \"09619584-4053-472a-9723-3dce678c9f4c\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") "
Oct 30 18:54:33.482402 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.482229    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09619584-4053-472a-9723-3dce678c9f4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09619584-4053-472a-9723-3dce678c9f4c-istio-podinfo\") pod \"09619584-4053-472a-9723-3dce678c9f4c\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") "
Oct 30 18:54:33.482402 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.482314    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-istio-envoy\") pod \"09619584-4053-472a-9723-3dce678c9f4c\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") "
Oct 30 18:54:33.482402 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.482389    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-istio-data" pod "09619584-4053-472a-9723-3dce678c9f4c" (UID: "09619584-4053-472a-9723-3dce678c9f4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:33.482686 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.482408    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-workload-socket" pod "09619584-4053-472a-9723-3dce678c9f4c" (UID: "09619584-4053-472a-9723-3dce678c9f4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:33.482686 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.482420    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-workload-certs\") pod \"09619584-4053-472a-9723-3dce678c9f4c\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") "
Oct 30 18:54:33.482686 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.482474    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-credential-socket" pod "09619584-4053-472a-9723-3dce678c9f4c" (UID: "09619584-4053-472a-9723-3dce678c9f4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:33.482686 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.482551    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09619584-4053-472a-9723-3dce678c9f4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09619584-4053-472a-9723-3dce678c9f4c-istiod-ca-cert\") pod \"09619584-4053-472a-9723-3dce678c9f4c\" (UID: \"09619584-4053-472a-9723-3dce678c9f4c\") "
Oct 30 18:54:33.482686 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.482586    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-workload-certs" pod "09619584-4053-472a-9723-3dce678c9f4c" (UID: "09619584-4053-472a-9723-3dce678c9f4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:33.483264 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.483235    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09619584-4053-472a-9723-3dce678c9f4c-istiod-ca-cert" pod "09619584-4053-472a-9723-3dce678c9f4c" (UID: "09619584-4053-472a-9723-3dce678c9f4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:33.483535 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.483513    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m5sj\" (UniqueName: \"kubernetes.io/projected/46273550-7eb9-47ba-8216-3d586b219ef3-kube-api-access-9m5sj\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.483633 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.483543    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46273550-7eb9-47ba-8216-3d586b219ef3-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.483633 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.483560    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.483633 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.483577    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46273550-7eb9-47ba-8216-3d586b219ef3-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.483633 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.483592    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.483633 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.483605    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46273550-7eb9-47ba-8216-3d586b219ef3-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.483633 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.483620    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.483633 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.483634    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.484878 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.484850    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09619584-4053-472a-9723-3dce678c9f4c-kube-api-access-jncbz" pod "09619584-4053-472a-9723-3dce678c9f4c" (UID: "09619584-4053-472a-9723-3dce678c9f4c"). InnerVolumeSpecName "kube-api-access-jncbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:33.485483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.485462    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09619584-4053-472a-9723-3dce678c9f4c-istio-token" pod "09619584-4053-472a-9723-3dce678c9f4c" (UID: "09619584-4053-472a-9723-3dce678c9f4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:33.485577 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.485564    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-istio-envoy" pod "09619584-4053-472a-9723-3dce678c9f4c" (UID: "09619584-4053-472a-9723-3dce678c9f4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:33.486355 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.486330    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09619584-4053-472a-9723-3dce678c9f4c-istio-podinfo" pod "09619584-4053-472a-9723-3dce678c9f4c" (UID: "09619584-4053-472a-9723-3dce678c9f4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:54:33.584953 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.584861    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09619584-4053-472a-9723-3dce678c9f4c-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.584953 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.584896    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jncbz\" (UniqueName: \"kubernetes.io/projected/09619584-4053-472a-9723-3dce678c9f4c-kube-api-access-jncbz\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.584953 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.584910    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09619584-4053-472a-9723-3dce678c9f4c-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.584953 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.584921    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09619584-4053-472a-9723-3dce678c9f4c-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.584953 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.584934    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09619584-4053-472a-9723-3dce678c9f4c-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:33.757858 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.757829    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"]
Oct 30 18:54:33.758542 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.758525    4372 cpu_manager.go:414] "containerMap: removing container" podUID="46273550-7eb9-47ba-8216-3d586b219ef3" containerName="istio-proxy"
Oct 30 18:54:33.758641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.758544    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46273550-7eb9-47ba-8216-3d586b219ef3" containerName="istio-proxy"
Oct 30 18:54:33.758641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.758570    4372 cpu_manager.go:414] "containerMap: removing container" podUID="09619584-4053-472a-9723-3dce678c9f4c" containerName="istio-proxy"
Oct 30 18:54:33.758641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.758575    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09619584-4053-472a-9723-3dce678c9f4c" containerName="istio-proxy"
Oct 30 18:54:33.758795 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.758681    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="46273550-7eb9-47ba-8216-3d586b219ef3" containerName="istio-proxy"
Oct 30 18:54:33.758795 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.758692    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="09619584-4053-472a-9723-3dce678c9f4c" containerName="istio-proxy"
Oct 30 18:54:33.775728 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.775705    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"]
Oct 30 18:54:33.776001 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.775980    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:33.781564 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.781461    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-fb19ff-serving-cert\""
Oct 30 18:54:33.889969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.889903    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-metrics-certs\") pod \"maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:33.889969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.889953    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cczc6\" (UniqueName: \"kubernetes.io/projected/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-kube-api-access-cczc6\") pod \"maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:33.890206 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.890167    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:33.890264 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:33.890221    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:34.010599 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.010563    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:34.010929 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.010904    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:34.012064 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.012040    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-metrics-certs\") pod \"maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:34.012337 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.012316    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cczc6\" (UniqueName: \"kubernetes.io/projected/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-kube-api-access-cczc6\") pod \"maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:34.012600 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.011895    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:34.025866 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.025837    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:34.029901 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.026021    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-metrics-certs\") pod \"maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:34.030185 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.026288    4372 generic.go:356] "Generic (PLEG): container finished" podID="46273550-7eb9-47ba-8216-3d586b219ef3" containerID="eda093a54590c6fe28ede98d7a26b376e130648381ad9bca5c82426a9af594e7" exitCode=0
Oct 30 18:54:34.030830 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.026315    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9" event={"ID":"46273550-7eb9-47ba-8216-3d586b219ef3","Type":"ContainerDied","Data":"eda093a54590c6fe28ede98d7a26b376e130648381ad9bca5c82426a9af594e7"}
Oct 30 18:54:34.030830 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.030598    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9" event={"ID":"46273550-7eb9-47ba-8216-3d586b219ef3","Type":"ContainerDied","Data":"632982e79f900b11e4febaaa59d39195a6eb4f147b12d7dbce64e619ef6ed082"}
Oct 30 18:54:34.030830 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.026389    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"
Oct 30 18:54:34.030830 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.030646    4372 scope.go:122] "RemoveContainer" containerID="eda093a54590c6fe28ede98d7a26b376e130648381ad9bca5c82426a9af594e7"
Oct 30 18:54:34.038154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.033177    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cczc6\" (UniqueName: \"kubernetes.io/projected/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-kube-api-access-cczc6\") pod \"maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:34.038154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.036367    4372 generic.go:356] "Generic (PLEG): container finished" podID="09619584-4053-472a-9723-3dce678c9f4c" containerID="e72e71e827125b14124aa108ec9c48dfed1e0869f0d39bfd2171f7681908fcca" exitCode=0
Oct 30 18:54:34.038154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.036420    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7" event={"ID":"09619584-4053-472a-9723-3dce678c9f4c","Type":"ContainerDied","Data":"e72e71e827125b14124aa108ec9c48dfed1e0869f0d39bfd2171f7681908fcca"}
Oct 30 18:54:34.038154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.036479    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7" event={"ID":"09619584-4053-472a-9723-3dce678c9f4c","Type":"ContainerDied","Data":"70bda33ca8ef84215d6a44eb343d515d2888fb42ddbe19319eb0781cd5076c92"}
Oct 30 18:54:34.038154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.036558    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"
Oct 30 18:54:34.051337 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.051316    4372 scope.go:122] "RemoveContainer" containerID="eda093a54590c6fe28ede98d7a26b376e130648381ad9bca5c82426a9af594e7"
Oct 30 18:54:34.051643 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:54:34.051625    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda093a54590c6fe28ede98d7a26b376e130648381ad9bca5c82426a9af594e7\": container with ID starting with eda093a54590c6fe28ede98d7a26b376e130648381ad9bca5c82426a9af594e7 not found: ID does not exist" containerID="eda093a54590c6fe28ede98d7a26b376e130648381ad9bca5c82426a9af594e7"
Oct 30 18:54:34.051706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.051651    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda093a54590c6fe28ede98d7a26b376e130648381ad9bca5c82426a9af594e7"} err="failed to get container status \"eda093a54590c6fe28ede98d7a26b376e130648381ad9bca5c82426a9af594e7\": rpc error: code = NotFound desc = could not find container \"eda093a54590c6fe28ede98d7a26b376e130648381ad9bca5c82426a9af594e7\": container with ID starting with eda093a54590c6fe28ede98d7a26b376e130648381ad9bca5c82426a9af594e7 not found: ID does not exist"
Oct 30 18:54:34.051706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.051666    4372 scope.go:122] "RemoveContainer" containerID="e72e71e827125b14124aa108ec9c48dfed1e0869f0d39bfd2171f7681908fcca"
Oct 30 18:54:34.060638 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.060594    4372 scope.go:122] "RemoveContainer" containerID="e72e71e827125b14124aa108ec9c48dfed1e0869f0d39bfd2171f7681908fcca"
Oct 30 18:54:34.060984 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:54:34.060964    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72e71e827125b14124aa108ec9c48dfed1e0869f0d39bfd2171f7681908fcca\": container with ID starting with e72e71e827125b14124aa108ec9c48dfed1e0869f0d39bfd2171f7681908fcca not found: ID does not exist" containerID="e72e71e827125b14124aa108ec9c48dfed1e0869f0d39bfd2171f7681908fcca"
Oct 30 18:54:34.061053 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.060991    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72e71e827125b14124aa108ec9c48dfed1e0869f0d39bfd2171f7681908fcca"} err="failed to get container status \"e72e71e827125b14124aa108ec9c48dfed1e0869f0d39bfd2171f7681908fcca\": rpc error: code = NotFound desc = could not find container \"e72e71e827125b14124aa108ec9c48dfed1e0869f0d39bfd2171f7681908fcca\": container with ID starting with e72e71e827125b14124aa108ec9c48dfed1e0869f0d39bfd2171f7681908fcca not found: ID does not exist"
Oct 30 18:54:34.065623 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.065603    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"]
Oct 30 18:54:34.078537 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.078516    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-fsvc9"]
Oct 30 18:54:34.091797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.091772    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"]
Oct 30 18:54:34.094565 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.094464    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:34.098448 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.098428    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-p7sd7"]
Oct 30 18:54:34.266426 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.266399    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09619584-4053-472a-9723-3dce678c9f4c" path="/var/lib/kubelet/pods/09619584-4053-472a-9723-3dce678c9f4c/volumes"
Oct 30 18:54:34.266889 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.266876    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46273550-7eb9-47ba-8216-3d586b219ef3" path="/var/lib/kubelet/pods/46273550-7eb9-47ba-8216-3d586b219ef3/volumes"
Oct 30 18:54:34.449234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:34.442261    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"]
Oct 30 18:54:35.047097 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:35.047066    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm" event={"ID":"77adbd2d-552e-4a78-a8b7-1ccf8db4a679","Type":"ContainerStarted","Data":"9d8f54b29a537cab8ca12da995dbafc91ef6d4d848dd1b0d312b8607229f5701"}
Oct 30 18:54:35.047247 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:35.047185    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm" event={"ID":"77adbd2d-552e-4a78-a8b7-1ccf8db4a679","Type":"ContainerStarted","Data":"e579d08e84032c283ca816c2134d0b909c660d34bc96e77f4f71887347cfcc00"}
Oct 30 18:54:35.047247 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:35.047233    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:35.070026 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:35.069984    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm" podStartSLOduration=1.612521519 podStartE2EDuration="2.069971142s" totalImagesPullingTime="457.449623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:35.067267657 +0000 UTC m=+1288.935685996" watchObservedRunningTime="2026-10-30 18:54:35.069971142 +0000 UTC m=+1288.938389472"
Oct 30 18:54:36.777919 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:36.777886    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:54:39.727253 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:39.727215    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"]
Oct 30 18:54:39.732967 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:39.732937    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:39.747457 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:39.747431    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"]
Oct 30 18:54:39.750027 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:39.749995    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-59f140-serving-cert\""
Oct 30 18:54:39.857723 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:39.857688    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06507e5a-e223-450e-8fb8-30b8d059aa04-metrics-certs\") pod \"maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:39.857977 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:39.857957    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06507e5a-e223-450e-8fb8-30b8d059aa04-maas-api-tls\") pod \"maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:39.858109 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:39.858087    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmc4\" (UniqueName: \"kubernetes.io/projected/06507e5a-e223-450e-8fb8-30b8d059aa04-kube-api-access-6bmc4\") pod \"maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:39.858248 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:39.858231    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06507e5a-e223-450e-8fb8-30b8d059aa04-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:39.993283 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:39.984794    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06507e5a-e223-450e-8fb8-30b8d059aa04-maas-api-tls\") pod \"maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:39.993283 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:39.984978    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmc4\" (UniqueName: \"kubernetes.io/projected/06507e5a-e223-450e-8fb8-30b8d059aa04-kube-api-access-6bmc4\") pod \"maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:39.993283 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:39.985061    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06507e5a-e223-450e-8fb8-30b8d059aa04-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:39.993283 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:39.985164    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06507e5a-e223-450e-8fb8-30b8d059aa04-metrics-certs\") pod \"maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:39.993609 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:39.993393    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06507e5a-e223-450e-8fb8-30b8d059aa04-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:40.011149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.002379    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06507e5a-e223-450e-8fb8-30b8d059aa04-metrics-certs\") pod \"maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:40.011149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.007080    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmc4\" (UniqueName: \"kubernetes.io/projected/06507e5a-e223-450e-8fb8-30b8d059aa04-kube-api-access-6bmc4\") pod \"maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:40.036149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.026733    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06507e5a-e223-450e-8fb8-30b8d059aa04-maas-api-tls\") pod \"maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:40.058606 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.051480    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:40.473885 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.473853    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"]
Oct 30 18:54:40.476457 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:54:40.476415    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06507e5a_e223_450e_8fb8_30b8d059aa04.slice/crio-59b643b62f68bbf6631c6aa7b072910cb47b404b1d71ec54b82ea9f4a0bfd083 WatchSource:0}: Error finding container 59b643b62f68bbf6631c6aa7b072910cb47b404b1d71ec54b82ea9f4a0bfd083: Status 404 returned error can't find the container with id 59b643b62f68bbf6631c6aa7b072910cb47b404b1d71ec54b82ea9f4a0bfd083
Oct 30 18:54:40.658539 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.658505    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"]
Oct 30 18:54:40.665487 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.665462    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:40.672807 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.672777    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-9d42ff-serving-cert\""
Oct 30 18:54:40.694309 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.678382    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"]
Oct 30 18:54:40.761643 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.761564    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825a5ded-653f-4172-9001-472d28b62d02-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:40.761643 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.761601    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzrhk\" (UniqueName: \"kubernetes.io/projected/825a5ded-653f-4172-9001-472d28b62d02-kube-api-access-qzrhk\") pod \"maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:40.762088 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.761700    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a5ded-653f-4172-9001-472d28b62d02-metrics-certs\") pod \"maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:40.762088 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.761805    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/825a5ded-653f-4172-9001-472d28b62d02-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:40.863040 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.863011    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a5ded-653f-4172-9001-472d28b62d02-metrics-certs\") pod \"maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:40.863259 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.863104    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/825a5ded-653f-4172-9001-472d28b62d02-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:40.863381 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.863363    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825a5ded-653f-4172-9001-472d28b62d02-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:40.863449 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.863397    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrhk\" (UniqueName: \"kubernetes.io/projected/825a5ded-653f-4172-9001-472d28b62d02-kube-api-access-qzrhk\") pod \"maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:40.863808 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.863788    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/825a5ded-653f-4172-9001-472d28b62d02-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:40.865657 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.865637    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a5ded-653f-4172-9001-472d28b62d02-metrics-certs\") pod \"maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:40.865762 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.865745    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825a5ded-653f-4172-9001-472d28b62d02-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:40.877485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:40.877462    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzrhk\" (UniqueName: \"kubernetes.io/projected/825a5ded-653f-4172-9001-472d28b62d02-kube-api-access-qzrhk\") pod \"maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:41.030303 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:41.023435    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:41.064243 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:41.064218    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:54:41.145317 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:41.145290    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97" event={"ID":"06507e5a-e223-450e-8fb8-30b8d059aa04","Type":"ContainerStarted","Data":"59b643b62f68bbf6631c6aa7b072910cb47b404b1d71ec54b82ea9f4a0bfd083"}
Oct 30 18:54:41.499630 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:41.493814    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"]
Oct 30 18:54:41.951966 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:41.951939    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"
Oct 30 18:54:41.952361 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:41.952031    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"
Oct 30 18:54:42.153295 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:42.153254    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97" event={"ID":"06507e5a-e223-450e-8fb8-30b8d059aa04","Type":"ContainerStarted","Data":"e7e4d048082885baa435d7e6a214972c444e311b24919acbdf4b10c0c229a5b6"}
Oct 30 18:54:42.153459 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:42.153345    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:42.154954 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:42.154928    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2" event={"ID":"825a5ded-653f-4172-9001-472d28b62d02","Type":"ContainerStarted","Data":"b4494e7ad13c1ca692b4d6567882af680a81fadb78264f8de3d0ea7224961470"}
Oct 30 18:54:42.155076 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:42.154972    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:42.155076 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:42.154988    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2" event={"ID":"825a5ded-653f-4172-9001-472d28b62d02","Type":"ContainerStarted","Data":"070e792b42699463c1feb069e768a4689d2a28453127222b868144265a5a3bf4"}
Oct 30 18:54:42.177556 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:42.177515    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97" podStartSLOduration=2.597605025 podStartE2EDuration="3.177503529s" totalImagesPullingTime="579.898504ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:42.175031208 +0000 UTC m=+1296.043449532" watchObservedRunningTime="2026-10-30 18:54:42.177503529 +0000 UTC m=+1296.045921859"
Oct 30 18:54:42.213678 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:42.213590    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2" podStartSLOduration=1.765095787 podStartE2EDuration="2.213576996s" totalImagesPullingTime="448.481209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:42.209301933 +0000 UTC m=+1296.077720264" watchObservedRunningTime="2026-10-30 18:54:42.213576996 +0000 UTC m=+1296.081995327"
Oct 30 18:54:48.131723 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:48.131692    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"]
Oct 30 18:54:48.132252 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:48.131932    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr" podUID="18afbe35-49b3-4670-b3a1-ac9a184eeb8d" containerName="istio-proxy" containerID="cri-o://ad1f3f0ff0e4de03fa05edc0560c665b903c810a1b035a805260454a423e4933" gracePeriod=30
Oct 30 18:54:48.135864 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:48.135838    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"]
Oct 30 18:54:48.136061 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:48.136040    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t" podUID="5567ca68-ab6f-496b-8538-c6102a04364c" containerName="istio-proxy" containerID="cri-o://bcd8f974ac4b3ffa15650dcd2d8ab25ce111b215a617713ba0d3dcee33ef400b" gracePeriod=30
Oct 30 18:54:48.169863 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:48.169835    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:54:48.177527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:48.177502    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:54:48.200935 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:48.200904    4372 generic.go:356] "Generic (PLEG): container finished" podID="15bc1bcf-0466-40c4-98af-928b2f3a259d" containerID="77ef934c9b05baf0b75ac32cc792f9209ad4805e53e7007016ece1e9d6f25808" exitCode=0
Oct 30 18:54:48.201060 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:48.200966    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk" event={"ID":"15bc1bcf-0466-40c4-98af-928b2f3a259d","Type":"ContainerDied","Data":"77ef934c9b05baf0b75ac32cc792f9209ad4805e53e7007016ece1e9d6f25808"}
Oct 30 18:54:48.201378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:48.201363    4372 scope.go:122] "RemoveContainer" containerID="77ef934c9b05baf0b75ac32cc792f9209ad4805e53e7007016ece1e9d6f25808"
Oct 30 18:54:48.202788 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:48.202757    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:54:48.202886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:48.202829    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:54:48.202886 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:48.202861    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:54:49.213138 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:49.213084    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk" event={"ID":"15bc1bcf-0466-40c4-98af-928b2f3a259d","Type":"ContainerStarted","Data":"29a271336ed2a33008b7c205aa861aa987c13614e640ddc795047d35b988f94b"}
Oct 30 18:54:50.055488 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:50.055459    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:54:50.055659 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:50.055585    4372 patch_prober.go:29] interesting pod/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 18:54:50.055659 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:50.055630    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk" podUID="15bc1bcf-0466-40c4-98af-928b2f3a259d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 18:54:51.055656 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:51.055622    4372 patch_prober.go:29] interesting pod/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 18:54:51.056072 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:51.055694    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk" podUID="15bc1bcf-0466-40c4-98af-928b2f3a259d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 18:54:52.054930 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:52.054898    4372 patch_prober.go:29] interesting pod/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 18:54:52.055105 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:52.054972    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk" podUID="15bc1bcf-0466-40c4-98af-928b2f3a259d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 18:54:53.055631 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.055593    4372 patch_prober.go:29] interesting pod/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 18:54:53.056180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.055659    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk" podUID="15bc1bcf-0466-40c4-98af-928b2f3a259d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 18:54:53.258157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.258112    4372 generic.go:356] "Generic (PLEG): container finished" podID="5567ca68-ab6f-496b-8538-c6102a04364c" containerID="bcd8f974ac4b3ffa15650dcd2d8ab25ce111b215a617713ba0d3dcee33ef400b" exitCode=0
Oct 30 18:54:53.258276 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.258206    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t" event={"ID":"5567ca68-ab6f-496b-8538-c6102a04364c","Type":"ContainerDied","Data":"bcd8f974ac4b3ffa15650dcd2d8ab25ce111b215a617713ba0d3dcee33ef400b"}
Oct 30 18:54:53.260217 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.260185    4372 generic.go:356] "Generic (PLEG): container finished" podID="18afbe35-49b3-4670-b3a1-ac9a184eeb8d" containerID="ad1f3f0ff0e4de03fa05edc0560c665b903c810a1b035a805260454a423e4933" exitCode=0
Oct 30 18:54:53.260345 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.260254    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr" event={"ID":"18afbe35-49b3-4670-b3a1-ac9a184eeb8d","Type":"ContainerDied","Data":"ad1f3f0ff0e4de03fa05edc0560c665b903c810a1b035a805260454a423e4933"}
Oct 30 18:54:53.774633 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.774604    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.821469    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-podinfo\") pod \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") "
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.821550    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-data\") pod \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") "
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.821615    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-envoy\") pod \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") "
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.821728    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-workload-socket\") pod \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") "
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.821816    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-kube-api-access-vvjm6\" (UniqueName: \"kubernetes.io/projected/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-kube-api-access-vvjm6\") pod \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") "
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.821847    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-token\" (UniqueName: \"kubernetes.io/projected/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-token\") pod \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") "
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.821901    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-workload-certs\") pod \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") "
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.821992    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-credential-socket\") pod \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") "
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.822034    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istiod-ca-cert\") pod \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\" (UID: \"18afbe35-49b3-4670-b3a1-ac9a184eeb8d\") "
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.822379    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-data" pod "18afbe35-49b3-4670-b3a1-ac9a184eeb8d" (UID: "18afbe35-49b3-4670-b3a1-ac9a184eeb8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.822690    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istiod-ca-cert" pod "18afbe35-49b3-4670-b3a1-ac9a184eeb8d" (UID: "18afbe35-49b3-4670-b3a1-ac9a184eeb8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.822871    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-workload-socket" pod "18afbe35-49b3-4670-b3a1-ac9a184eeb8d" (UID: "18afbe35-49b3-4670-b3a1-ac9a184eeb8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.822973    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.822995    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.822997    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-credential-socket" pod "18afbe35-49b3-4670-b3a1-ac9a184eeb8d" (UID: "18afbe35-49b3-4670-b3a1-ac9a184eeb8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.823080    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-workload-certs" pod "18afbe35-49b3-4670-b3a1-ac9a184eeb8d" (UID: "18afbe35-49b3-4670-b3a1-ac9a184eeb8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.824488    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-podinfo" pod "18afbe35-49b3-4670-b3a1-ac9a184eeb8d" (UID: "18afbe35-49b3-4670-b3a1-ac9a184eeb8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.824513    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-envoy" pod "18afbe35-49b3-4670-b3a1-ac9a184eeb8d" (UID: "18afbe35-49b3-4670-b3a1-ac9a184eeb8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:53.826411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.825225    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-kube-api-access-vvjm6" pod "18afbe35-49b3-4670-b3a1-ac9a184eeb8d" (UID: "18afbe35-49b3-4670-b3a1-ac9a184eeb8d"). InnerVolumeSpecName "kube-api-access-vvjm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:53.828516 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.828334    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-token" pod "18afbe35-49b3-4670-b3a1-ac9a184eeb8d" (UID: "18afbe35-49b3-4670-b3a1-ac9a184eeb8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:53.848872 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.848852    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:54:53.923604 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.923575    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-istio-envoy\") pod \"5567ca68-ab6f-496b-8538-c6102a04364c\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") "
Oct 30 18:54:53.923772 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.923678    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-workload-certs\") pod \"5567ca68-ab6f-496b-8538-c6102a04364c\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") "
Oct 30 18:54:53.923844 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.923826    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-istio-data\") pod \"5567ca68-ab6f-496b-8538-c6102a04364c\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") "
Oct 30 18:54:53.923888 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.923872    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-workload-socket\") pod \"5567ca68-ab6f-496b-8538-c6102a04364c\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") "
Oct 30 18:54:53.923958 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.923916    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5567ca68-ab6f-496b-8538-c6102a04364c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5567ca68-ab6f-496b-8538-c6102a04364c-istio-podinfo\") pod \"5567ca68-ab6f-496b-8538-c6102a04364c\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") "
Oct 30 18:54:53.924020 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.923988    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5567ca68-ab6f-496b-8538-c6102a04364c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5567ca68-ab6f-496b-8538-c6102a04364c-istiod-ca-cert\") pod \"5567ca68-ab6f-496b-8538-c6102a04364c\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") "
Oct 30 18:54:53.924105 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.924087    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5567ca68-ab6f-496b-8538-c6102a04364c-kube-api-access-hlc6t\" (UniqueName: \"kubernetes.io/projected/5567ca68-ab6f-496b-8538-c6102a04364c-kube-api-access-hlc6t\") pod \"5567ca68-ab6f-496b-8538-c6102a04364c\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") "
Oct 30 18:54:53.924205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.924113    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-workload-socket" pod "5567ca68-ab6f-496b-8538-c6102a04364c" (UID: "5567ca68-ab6f-496b-8538-c6102a04364c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:53.924205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.924103    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-istio-data" pod "5567ca68-ab6f-496b-8538-c6102a04364c" (UID: "5567ca68-ab6f-496b-8538-c6102a04364c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:53.924527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.924203    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5567ca68-ab6f-496b-8538-c6102a04364c-istio-token\" (UniqueName: \"kubernetes.io/projected/5567ca68-ab6f-496b-8538-c6102a04364c-istio-token\") pod \"5567ca68-ab6f-496b-8538-c6102a04364c\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") "
Oct 30 18:54:53.924527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.924244    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-workload-certs" pod "5567ca68-ab6f-496b-8538-c6102a04364c" (UID: "5567ca68-ab6f-496b-8538-c6102a04364c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:53.924527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.924245    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-credential-socket\") pod \"5567ca68-ab6f-496b-8538-c6102a04364c\" (UID: \"5567ca68-ab6f-496b-8538-c6102a04364c\") "
Oct 30 18:54:53.926405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.924373    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5567ca68-ab6f-496b-8538-c6102a04364c-istiod-ca-cert" pod "5567ca68-ab6f-496b-8538-c6102a04364c" (UID: "5567ca68-ab6f-496b-8538-c6102a04364c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:53.927662 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.927632    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-credential-socket" pod "5567ca68-ab6f-496b-8538-c6102a04364c" (UID: "5567ca68-ab6f-496b-8538-c6102a04364c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:53.928026 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.927963    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:53.928026 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.927993    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:53.928440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.928160    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:53.928440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.928183    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:53.928440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.928205    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:53.928440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.928330    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:53.928440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.928346    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:53.928440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.928362    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:53.929770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.928519    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5567ca68-ab6f-496b-8538-c6102a04364c-istio-podinfo" pod "5567ca68-ab6f-496b-8538-c6102a04364c" (UID: "5567ca68-ab6f-496b-8538-c6102a04364c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:54:53.929770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.928487    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5567ca68-ab6f-496b-8538-c6102a04364c-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:53.929770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.928654    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvjm6\" (UniqueName: \"kubernetes.io/projected/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-kube-api-access-vvjm6\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:53.929770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.928670    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18afbe35-49b3-4670-b3a1-ac9a184eeb8d-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:53.929770 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.929690    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5567ca68-ab6f-496b-8538-c6102a04364c-istio-token" pod "5567ca68-ab6f-496b-8538-c6102a04364c" (UID: "5567ca68-ab6f-496b-8538-c6102a04364c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:53.935278 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.935253    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-istio-envoy" pod "5567ca68-ab6f-496b-8538-c6102a04364c" (UID: "5567ca68-ab6f-496b-8538-c6102a04364c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:53.942250 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:53.937985    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5567ca68-ab6f-496b-8538-c6102a04364c-kube-api-access-hlc6t" pod "5567ca68-ab6f-496b-8538-c6102a04364c" (UID: "5567ca68-ab6f-496b-8538-c6102a04364c"). InnerVolumeSpecName "kube-api-access-hlc6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:54.030453 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.030377    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5567ca68-ab6f-496b-8538-c6102a04364c-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:54.030453 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.030406    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlc6t\" (UniqueName: \"kubernetes.io/projected/5567ca68-ab6f-496b-8538-c6102a04364c-kube-api-access-hlc6t\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:54.030453 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.030416    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5567ca68-ab6f-496b-8538-c6102a04364c-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:54.030453 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.030425    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:54.030453 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.030434    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5567ca68-ab6f-496b-8538-c6102a04364c-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:54.055588 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.055560    4372 patch_prober.go:29] interesting pod/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 18:54:54.055723 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.055614    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk" podUID="15bc1bcf-0466-40c4-98af-928b2f3a259d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 18:54:54.270850 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.270753    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t" event={"ID":"5567ca68-ab6f-496b-8538-c6102a04364c","Type":"ContainerDied","Data":"62c7dc6062fc7ba29a0f8bd6b3c4d149ef51326890dedac80ea27b068aba9072"}
Oct 30 18:54:54.270850 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.270830    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"
Oct 30 18:54:54.270850 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.270847    4372 scope.go:122] "RemoveContainer" containerID="bcd8f974ac4b3ffa15650dcd2d8ab25ce111b215a617713ba0d3dcee33ef400b"
Oct 30 18:54:54.272614 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.272584    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr" event={"ID":"18afbe35-49b3-4670-b3a1-ac9a184eeb8d","Type":"ContainerDied","Data":"d36954780969e1e6d64ec6fa2381671234e695512120f07165208a5989b97b05"}
Oct 30 18:54:54.272712 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.272684    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"
Oct 30 18:54:54.283671 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.283652    4372 scope.go:122] "RemoveContainer" containerID="ad1f3f0ff0e4de03fa05edc0560c665b903c810a1b035a805260454a423e4933"
Oct 30 18:54:54.305371 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.305350    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"]
Oct 30 18:54:54.309980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.309956    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-lvb8t"]
Oct 30 18:54:54.328493 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.328473    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"]
Oct 30 18:54:54.331592 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:54.331574    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-hpxkr"]
Oct 30 18:54:55.062449 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:55.062422    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:54:55.062874 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:55.062800    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:54:55.063397 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:55.063377    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:54:56.266698 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:56.266665    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18afbe35-49b3-4670-b3a1-ac9a184eeb8d" path="/var/lib/kubelet/pods/18afbe35-49b3-4670-b3a1-ac9a184eeb8d/volumes"
Oct 30 18:54:56.267387 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:56.267366    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5567ca68-ab6f-496b-8538-c6102a04364c" path="/var/lib/kubelet/pods/5567ca68-ab6f-496b-8538-c6102a04364c/volumes"
Oct 30 18:54:57.230321 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.230290    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"]
Oct 30 18:54:57.230925 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.230911    4372 cpu_manager.go:414] "containerMap: removing container" podUID="18afbe35-49b3-4670-b3a1-ac9a184eeb8d" containerName="istio-proxy"
Oct 30 18:54:57.230981 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.230927    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18afbe35-49b3-4670-b3a1-ac9a184eeb8d" containerName="istio-proxy"
Oct 30 18:54:57.230981 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.230969    4372 cpu_manager.go:414] "containerMap: removing container" podUID="5567ca68-ab6f-496b-8538-c6102a04364c" containerName="istio-proxy"
Oct 30 18:54:57.230981 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.230974    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5567ca68-ab6f-496b-8538-c6102a04364c" containerName="istio-proxy"
Oct 30 18:54:57.231085 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.231063    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="5567ca68-ab6f-496b-8538-c6102a04364c" containerName="istio-proxy"
Oct 30 18:54:57.231085 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.231073    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="18afbe35-49b3-4670-b3a1-ac9a184eeb8d" containerName="istio-proxy"
Oct 30 18:54:57.235748 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.235729    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.250668 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.250643    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"]
Oct 30 18:54:57.255989 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.255886    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-fa7b20\"/\"default-dockercfg-ct6fr\""
Oct 30 18:54:57.255989 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.255901    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-fa7b20\"/\"openshift-service-ca.crt\""
Oct 30 18:54:57.255989 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.255906    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-fa7b20\"/\"kube-root-ca.crt\""
Oct 30 18:54:57.255989 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.255901    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-fa7b20\"/\"praxis-route-fa7b20-kserve-self-signed-certs\""
Oct 30 18:54:57.286658 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.286628    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrltq\" (UniqueName: \"kubernetes.io/projected/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-kube-api-access-wrltq\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.286967 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.286670    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-home\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.286967 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.286773    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-model-cache\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.286967 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.286850    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-tls-certs\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.287074 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.287023    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-dshm\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.287106 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.287079    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-tmp-dir\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.388853 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.388831    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-model-cache\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.388964 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.388877    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-tls-certs\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.388964 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.388929    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-dshm\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.388964 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.388954    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-tmp-dir\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.389159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.389043    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrltq\" (UniqueName: \"kubernetes.io/projected/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-kube-api-access-wrltq\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.389159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.389065    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-home\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.389296 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.389274    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-model-cache\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.389359 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.389337    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-tmp-dir\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.389444 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.389423    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-home\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.391401 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.391381    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-dshm\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.391505 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.391485    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-tls-certs\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.401675 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.401656    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrltq\" (UniqueName: \"kubernetes.io/projected/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-kube-api-access-wrltq\") pod \"praxis-route-fa7b20-kserve-67df94c59c-fctbn\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") " pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.549201 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.548847    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:54:57.698803 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:54:57.698763    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:54:57.902319 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:54:57.902283    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ae2b494_2225_49bb_a1a5_2e45bc8cc12d.slice/crio-31a99fe44e722acfc98de205d3a78bda3715368c3f3846fba698bafce6750b26 WatchSource:0}: Error finding container 31a99fe44e722acfc98de205d3a78bda3715368c3f3846fba698bafce6750b26: Status 404 returned error can't find the container with id 31a99fe44e722acfc98de205d3a78bda3715368c3f3846fba698bafce6750b26
Oct 30 18:54:57.902485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:57.902346    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"]
Oct 30 18:54:58.304890 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:58.304794    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" event={"ID":"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d","Type":"ContainerStarted","Data":"11749201dd156a1bb61e77d95e7d646c690cfb775380a42261d8ce690ab8c7f7"}
Oct 30 18:54:58.304890 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:58.304850    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" event={"ID":"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d","Type":"ContainerStarted","Data":"31a99fe44e722acfc98de205d3a78bda3715368c3f3846fba698bafce6750b26"}
Oct 30 18:54:58.332345 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:58.332295    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" podStartSLOduration=1.3322814379999999 podStartE2EDuration="1.332281438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:58.330293737 +0000 UTC m=+1312.198712068" watchObservedRunningTime="2026-10-30 18:54:58.332281438 +0000 UTC m=+1312.200699767"
Oct 30 18:54:58.549389 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:54:58.549342    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:55:07.549697 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:07.549654    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:55:07.566253 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:07.566225    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:55:08.137018 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:08.136985    4372 scope.go:122] "RemoveContainer" containerID="dab16fb914a8a07190bafcb38a2ff335b656e561bcf4e89465d0df530e457141"
Oct 30 18:55:08.508896 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:08.508812    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:55:10.995953 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:10.995899    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"]
Oct 30 18:55:12.119144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:12.119106    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"]
Oct 30 18:55:12.132725 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:12.132693    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"]
Oct 30 18:55:12.537572 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:12.537481    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"]
Oct 30 18:55:12.573187 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:12.567517    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4" podUID="f6846d27-0053-4c6b-810a-1eab95f42638" containerName="maas-api" containerID="cri-o://c340bb6dedd27576066b73f3e219a3d18613215f50f6600531ab087959cc8eec" gracePeriod=30
Oct 30 18:55:12.978940 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:12.978224    4372 patch_prober.go:29] interesting pod/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:55:12.978940 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:12.978288    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh" podUID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:55:13.244264 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.244215    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:55:13.250101 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.250065    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6846d27-0053-4c6b-810a-1eab95f42638-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6846d27-0053-4c6b-810a-1eab95f42638-openshift-service-ca\") pod \"f6846d27-0053-4c6b-810a-1eab95f42638\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") "
Oct 30 18:55:13.250468 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.250446    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6846d27-0053-4c6b-810a-1eab95f42638-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6846d27-0053-4c6b-810a-1eab95f42638-metrics-certs\") pod \"f6846d27-0053-4c6b-810a-1eab95f42638\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") "
Oct 30 18:55:13.251021 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.250694    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6846d27-0053-4c6b-810a-1eab95f42638-openshift-service-ca" pod "f6846d27-0053-4c6b-810a-1eab95f42638" (UID: "f6846d27-0053-4c6b-810a-1eab95f42638"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:13.251164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.251005    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6846d27-0053-4c6b-810a-1eab95f42638-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6846d27-0053-4c6b-810a-1eab95f42638-maas-api-tls\") pod \"f6846d27-0053-4c6b-810a-1eab95f42638\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") "
Oct 30 18:55:13.251421 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.251401    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6846d27-0053-4c6b-810a-1eab95f42638-kube-api-access-wdxsz\" (UniqueName: \"kubernetes.io/projected/f6846d27-0053-4c6b-810a-1eab95f42638-kube-api-access-wdxsz\") pod \"f6846d27-0053-4c6b-810a-1eab95f42638\" (UID: \"f6846d27-0053-4c6b-810a-1eab95f42638\") "
Oct 30 18:55:13.252226 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.252205    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6846d27-0053-4c6b-810a-1eab95f42638-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:13.259434 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.259403    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6846d27-0053-4c6b-810a-1eab95f42638-maas-api-tls" pod "f6846d27-0053-4c6b-810a-1eab95f42638" (UID: "f6846d27-0053-4c6b-810a-1eab95f42638"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:13.261476 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.261428    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6846d27-0053-4c6b-810a-1eab95f42638-kube-api-access-wdxsz" pod "f6846d27-0053-4c6b-810a-1eab95f42638" (UID: "f6846d27-0053-4c6b-810a-1eab95f42638"). InnerVolumeSpecName "kube-api-access-wdxsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:13.276403 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.275022    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6846d27-0053-4c6b-810a-1eab95f42638-metrics-certs" pod "f6846d27-0053-4c6b-810a-1eab95f42638" (UID: "f6846d27-0053-4c6b-810a-1eab95f42638"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:13.353819 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.353737    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6846d27-0053-4c6b-810a-1eab95f42638-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:13.353819 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.353773    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdxsz\" (UniqueName: \"kubernetes.io/projected/f6846d27-0053-4c6b-810a-1eab95f42638-kube-api-access-wdxsz\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:13.353819 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.353791    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6846d27-0053-4c6b-810a-1eab95f42638-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:13.616837 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.616760    4372 generic.go:356] "Generic (PLEG): container finished" podID="f6846d27-0053-4c6b-810a-1eab95f42638" containerID="c340bb6dedd27576066b73f3e219a3d18613215f50f6600531ab087959cc8eec" exitCode=0
Oct 30 18:55:13.617014 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.616837    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"
Oct 30 18:55:13.617014 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.616855    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4" event={"ID":"f6846d27-0053-4c6b-810a-1eab95f42638","Type":"ContainerDied","Data":"c340bb6dedd27576066b73f3e219a3d18613215f50f6600531ab087959cc8eec"}
Oct 30 18:55:13.617014 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.616953    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4" event={"ID":"f6846d27-0053-4c6b-810a-1eab95f42638","Type":"ContainerDied","Data":"53dcf0a3a24418e7ddc498179ae7714c9216c5180b552a4b226bdb3b4509a074"}
Oct 30 18:55:13.617014 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.616994    4372 scope.go:122] "RemoveContainer" containerID="c340bb6dedd27576066b73f3e219a3d18613215f50f6600531ab087959cc8eec"
Oct 30 18:55:13.630165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.630146    4372 scope.go:122] "RemoveContainer" containerID="c340bb6dedd27576066b73f3e219a3d18613215f50f6600531ab087959cc8eec"
Oct 30 18:55:13.630449 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:55:13.630422    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c340bb6dedd27576066b73f3e219a3d18613215f50f6600531ab087959cc8eec\": container with ID starting with c340bb6dedd27576066b73f3e219a3d18613215f50f6600531ab087959cc8eec not found: ID does not exist" containerID="c340bb6dedd27576066b73f3e219a3d18613215f50f6600531ab087959cc8eec"
Oct 30 18:55:13.630578 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.630457    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c340bb6dedd27576066b73f3e219a3d18613215f50f6600531ab087959cc8eec"} err="failed to get container status \"c340bb6dedd27576066b73f3e219a3d18613215f50f6600531ab087959cc8eec\": rpc error: code = NotFound desc = could not find container \"c340bb6dedd27576066b73f3e219a3d18613215f50f6600531ab087959cc8eec\": container with ID starting with c340bb6dedd27576066b73f3e219a3d18613215f50f6600531ab087959cc8eec not found: ID does not exist"
Oct 30 18:55:13.648071 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.648049    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"]
Oct 30 18:55:13.653802 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:13.653781    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5ab51-7858f98cfc-lcss4"]
Oct 30 18:55:14.277475 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:14.277382    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6846d27-0053-4c6b-810a-1eab95f42638" path="/var/lib/kubelet/pods/f6846d27-0053-4c6b-810a-1eab95f42638/volumes"
Oct 30 18:55:14.625875 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:14.625843    4372 generic.go:356] "Generic (PLEG): container finished" podID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" containerID="531c3282d0ffad66fa9b62638c247e4d7124c4b1ecb08c3438e4c5936fabef36" exitCode=0
Oct 30 18:55:14.626058 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:14.625920    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh" event={"ID":"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4","Type":"ContainerDied","Data":"531c3282d0ffad66fa9b62638c247e4d7124c4b1ecb08c3438e4c5936fabef36"}
Oct 30 18:55:14.626367 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:14.626348    4372 scope.go:122] "RemoveContainer" containerID="531c3282d0ffad66fa9b62638c247e4d7124c4b1ecb08c3438e4c5936fabef36"
Oct 30 18:55:14.628088 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:14.628053    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:55:14.628180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:14.628164    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:55:14.628230 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:14.628215    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:55:15.636319 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:15.636280    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh" event={"ID":"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4","Type":"ContainerStarted","Data":"908f1dbf6602584fd0ded7e1e1a45049b2c2f5664c3e2c7f1d193cad6d0603f2"}
Oct 30 18:55:16.000294 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.000188    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95" podUID="55e51e05-f986-45b9-836d-6339a685318d" containerName="payload-processing" containerID="cri-o://0056289b238e757598abbe59e9f05cab3d0e18db0a96a3f933477f50dc750b5f" gracePeriod=25
Oct 30 18:55:16.205290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.193520    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:55:16.205290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.193640    4372 patch_prober.go:29] interesting pod/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:55:16.205290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.193680    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh" podUID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 18:55:16.335029 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.335005    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"
Oct 30 18:55:16.415899 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.415861    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e51e05-f986-45b9-836d-6339a685318d-kube-api-access-5slpt\" (UniqueName: \"kubernetes.io/projected/55e51e05-f986-45b9-836d-6339a685318d-kube-api-access-5slpt\") pod \"55e51e05-f986-45b9-836d-6339a685318d\" (UID: \"55e51e05-f986-45b9-836d-6339a685318d\") "
Oct 30 18:55:16.416083 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.416036    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e51e05-f986-45b9-836d-6339a685318d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55e51e05-f986-45b9-836d-6339a685318d-plugins-config-volume\") pod \"55e51e05-f986-45b9-836d-6339a685318d\" (UID: \"55e51e05-f986-45b9-836d-6339a685318d\") "
Oct 30 18:55:16.416527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.416496    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e51e05-f986-45b9-836d-6339a685318d-plugins-config-volume" pod "55e51e05-f986-45b9-836d-6339a685318d" (UID: "55e51e05-f986-45b9-836d-6339a685318d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:16.418347 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.418320    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e51e05-f986-45b9-836d-6339a685318d-kube-api-access-5slpt" pod "55e51e05-f986-45b9-836d-6339a685318d" (UID: "55e51e05-f986-45b9-836d-6339a685318d"). InnerVolumeSpecName "kube-api-access-5slpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:16.517572 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.517535    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55e51e05-f986-45b9-836d-6339a685318d-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:16.517572 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.517562    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5slpt\" (UniqueName: \"kubernetes.io/projected/55e51e05-f986-45b9-836d-6339a685318d-kube-api-access-5slpt\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:16.663162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.662218    4372 generic.go:356] "Generic (PLEG): container finished" podID="55e51e05-f986-45b9-836d-6339a685318d" containerID="0056289b238e757598abbe59e9f05cab3d0e18db0a96a3f933477f50dc750b5f" exitCode=0
Oct 30 18:55:16.663712 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.663439    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"
Oct 30 18:55:16.668244 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.667226    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95" event={"ID":"55e51e05-f986-45b9-836d-6339a685318d","Type":"ContainerDied","Data":"0056289b238e757598abbe59e9f05cab3d0e18db0a96a3f933477f50dc750b5f"}
Oct 30 18:55:16.668244 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.667299    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95" event={"ID":"55e51e05-f986-45b9-836d-6339a685318d","Type":"ContainerDied","Data":"d327654a3ec4c1f98566412af1062ce4c5712f6e6b231ce9b3939a1f49d0324d"}
Oct 30 18:55:16.668244 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.667336    4372 scope.go:122] "RemoveContainer" containerID="0056289b238e757598abbe59e9f05cab3d0e18db0a96a3f933477f50dc750b5f"
Oct 30 18:55:16.710329 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.710270    4372 scope.go:122] "RemoveContainer" containerID="0056289b238e757598abbe59e9f05cab3d0e18db0a96a3f933477f50dc750b5f"
Oct 30 18:55:16.710803 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:55:16.710737    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0056289b238e757598abbe59e9f05cab3d0e18db0a96a3f933477f50dc750b5f\": container with ID starting with 0056289b238e757598abbe59e9f05cab3d0e18db0a96a3f933477f50dc750b5f not found: ID does not exist" containerID="0056289b238e757598abbe59e9f05cab3d0e18db0a96a3f933477f50dc750b5f"
Oct 30 18:55:16.710803 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.710767    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0056289b238e757598abbe59e9f05cab3d0e18db0a96a3f933477f50dc750b5f"} err="failed to get container status \"0056289b238e757598abbe59e9f05cab3d0e18db0a96a3f933477f50dc750b5f\": rpc error: code = NotFound desc = could not find container \"0056289b238e757598abbe59e9f05cab3d0e18db0a96a3f933477f50dc750b5f\": container with ID starting with 0056289b238e757598abbe59e9f05cab3d0e18db0a96a3f933477f50dc750b5f not found: ID does not exist"
Oct 30 18:55:16.711113 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.711098    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"]
Oct 30 18:55:16.714608 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:16.714586    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5ab51-586bc48788-pbm95"]
Oct 30 18:55:17.119828 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.119789    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98" podUID="1d5c2659-c280-40fb-b2da-bf7672b55833" containerName="payload-processing" containerID="cri-o://7b9d4bf8b57eb9485642fdb601f6a81b27ee9ed2ad02b37ba91078ea7fac5fed" gracePeriod=25
Oct 30 18:55:17.133396 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.133363    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99" podUID="9c35fda7-eeab-45c3-9fdc-fc00bc12f141" containerName="payload-pre-processing" containerID="cri-o://a991aa4040b1633a35b5e230260cc5378751cc11619b6b68be9d22a2c0d7df45" gracePeriod=25
Oct 30 18:55:17.196900 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.196462    4372 patch_prober.go:29] interesting pod/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:55:17.196900 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.196519    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh" podUID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 18:55:17.563404 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.563375    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"
Oct 30 18:55:17.657638 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.657611    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d5c2659-c280-40fb-b2da-bf7672b55833-kube-api-access-nk25f\" (UniqueName: \"kubernetes.io/projected/1d5c2659-c280-40fb-b2da-bf7672b55833-kube-api-access-nk25f\") pod \"1d5c2659-c280-40fb-b2da-bf7672b55833\" (UID: \"1d5c2659-c280-40fb-b2da-bf7672b55833\") "
Oct 30 18:55:17.657758 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.657739    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d5c2659-c280-40fb-b2da-bf7672b55833-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5c2659-c280-40fb-b2da-bf7672b55833-plugins-config-volume\") pod \"1d5c2659-c280-40fb-b2da-bf7672b55833\" (UID: \"1d5c2659-c280-40fb-b2da-bf7672b55833\") "
Oct 30 18:55:17.658078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.658052    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5c2659-c280-40fb-b2da-bf7672b55833-plugins-config-volume" pod "1d5c2659-c280-40fb-b2da-bf7672b55833" (UID: "1d5c2659-c280-40fb-b2da-bf7672b55833"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:17.658463 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.658443    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5c2659-c280-40fb-b2da-bf7672b55833-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:17.659829 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.659807    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5c2659-c280-40fb-b2da-bf7672b55833-kube-api-access-nk25f" pod "1d5c2659-c280-40fb-b2da-bf7672b55833" (UID: "1d5c2659-c280-40fb-b2da-bf7672b55833"). InnerVolumeSpecName "kube-api-access-nk25f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:17.663569 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.663526    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"
Oct 30 18:55:17.670583 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.670563    4372 generic.go:356] "Generic (PLEG): container finished" podID="9c35fda7-eeab-45c3-9fdc-fc00bc12f141" containerID="a991aa4040b1633a35b5e230260cc5378751cc11619b6b68be9d22a2c0d7df45" exitCode=0
Oct 30 18:55:17.670684 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.670606    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"
Oct 30 18:55:17.670684 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.670637    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99" event={"ID":"9c35fda7-eeab-45c3-9fdc-fc00bc12f141","Type":"ContainerDied","Data":"a991aa4040b1633a35b5e230260cc5378751cc11619b6b68be9d22a2c0d7df45"}
Oct 30 18:55:17.670799 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.670689    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99" event={"ID":"9c35fda7-eeab-45c3-9fdc-fc00bc12f141","Type":"ContainerDied","Data":"c1a919c6d565122241c02d1308a7abe8536fd5c9da7bd19303910765e1dd71da"}
Oct 30 18:55:17.670799 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.670719    4372 scope.go:122] "RemoveContainer" containerID="a991aa4040b1633a35b5e230260cc5378751cc11619b6b68be9d22a2c0d7df45"
Oct 30 18:55:17.673268 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.673249    4372 generic.go:356] "Generic (PLEG): container finished" podID="1d5c2659-c280-40fb-b2da-bf7672b55833" containerID="7b9d4bf8b57eb9485642fdb601f6a81b27ee9ed2ad02b37ba91078ea7fac5fed" exitCode=0
Oct 30 18:55:17.673354 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.673297    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98" event={"ID":"1d5c2659-c280-40fb-b2da-bf7672b55833","Type":"ContainerDied","Data":"7b9d4bf8b57eb9485642fdb601f6a81b27ee9ed2ad02b37ba91078ea7fac5fed"}
Oct 30 18:55:17.673354 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.673330    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98" event={"ID":"1d5c2659-c280-40fb-b2da-bf7672b55833","Type":"ContainerDied","Data":"720f4f35c2c2122f51fc6aa082d16c565447f7b916e648dfd84e48f746c3fa55"}
Oct 30 18:55:17.673436 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.673354    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"
Oct 30 18:55:17.711554 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.711332    4372 scope.go:122] "RemoveContainer" containerID="a991aa4040b1633a35b5e230260cc5378751cc11619b6b68be9d22a2c0d7df45"
Oct 30 18:55:17.711759 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:55:17.711732    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a991aa4040b1633a35b5e230260cc5378751cc11619b6b68be9d22a2c0d7df45\": container with ID starting with a991aa4040b1633a35b5e230260cc5378751cc11619b6b68be9d22a2c0d7df45 not found: ID does not exist" containerID="a991aa4040b1633a35b5e230260cc5378751cc11619b6b68be9d22a2c0d7df45"
Oct 30 18:55:17.711891 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.711773    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a991aa4040b1633a35b5e230260cc5378751cc11619b6b68be9d22a2c0d7df45"} err="failed to get container status \"a991aa4040b1633a35b5e230260cc5378751cc11619b6b68be9d22a2c0d7df45\": rpc error: code = NotFound desc = could not find container \"a991aa4040b1633a35b5e230260cc5378751cc11619b6b68be9d22a2c0d7df45\": container with ID starting with a991aa4040b1633a35b5e230260cc5378751cc11619b6b68be9d22a2c0d7df45 not found: ID does not exist"
Oct 30 18:55:17.711891 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.711797    4372 scope.go:122] "RemoveContainer" containerID="7b9d4bf8b57eb9485642fdb601f6a81b27ee9ed2ad02b37ba91078ea7fac5fed"
Oct 30 18:55:17.724528 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.722579    4372 scope.go:122] "RemoveContainer" containerID="7b9d4bf8b57eb9485642fdb601f6a81b27ee9ed2ad02b37ba91078ea7fac5fed"
Oct 30 18:55:17.724528 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:55:17.722876    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9d4bf8b57eb9485642fdb601f6a81b27ee9ed2ad02b37ba91078ea7fac5fed\": container with ID starting with 7b9d4bf8b57eb9485642fdb601f6a81b27ee9ed2ad02b37ba91078ea7fac5fed not found: ID does not exist" containerID="7b9d4bf8b57eb9485642fdb601f6a81b27ee9ed2ad02b37ba91078ea7fac5fed"
Oct 30 18:55:17.724528 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.722908    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9d4bf8b57eb9485642fdb601f6a81b27ee9ed2ad02b37ba91078ea7fac5fed"} err="failed to get container status \"7b9d4bf8b57eb9485642fdb601f6a81b27ee9ed2ad02b37ba91078ea7fac5fed\": rpc error: code = NotFound desc = could not find container \"7b9d4bf8b57eb9485642fdb601f6a81b27ee9ed2ad02b37ba91078ea7fac5fed\": container with ID starting with 7b9d4bf8b57eb9485642fdb601f6a81b27ee9ed2ad02b37ba91078ea7fac5fed not found: ID does not exist"
Oct 30 18:55:17.731309 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.731288    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"]
Oct 30 18:55:17.737283 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.737252    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f2ceb5-86487779c8-xws98"]
Oct 30 18:55:17.765559 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.765535    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c35fda7-eeab-45c3-9fdc-fc00bc12f141-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c35fda7-eeab-45c3-9fdc-fc00bc12f141-plugins-config-volume\") pod \"9c35fda7-eeab-45c3-9fdc-fc00bc12f141\" (UID: \"9c35fda7-eeab-45c3-9fdc-fc00bc12f141\") "
Oct 30 18:55:17.765700 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.765642    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c35fda7-eeab-45c3-9fdc-fc00bc12f141-kube-api-access-zgfp9\" (UniqueName: \"kubernetes.io/projected/9c35fda7-eeab-45c3-9fdc-fc00bc12f141-kube-api-access-zgfp9\") pod \"9c35fda7-eeab-45c3-9fdc-fc00bc12f141\" (UID: \"9c35fda7-eeab-45c3-9fdc-fc00bc12f141\") "
Oct 30 18:55:17.765917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.765893    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c35fda7-eeab-45c3-9fdc-fc00bc12f141-plugins-config-volume" pod "9c35fda7-eeab-45c3-9fdc-fc00bc12f141" (UID: "9c35fda7-eeab-45c3-9fdc-fc00bc12f141"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:17.766237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.766219    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk25f\" (UniqueName: \"kubernetes.io/projected/1d5c2659-c280-40fb-b2da-bf7672b55833-kube-api-access-nk25f\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:17.766237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.766238    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c35fda7-eeab-45c3-9fdc-fc00bc12f141-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:17.768106 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.768083    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c35fda7-eeab-45c3-9fdc-fc00bc12f141-kube-api-access-zgfp9" pod "9c35fda7-eeab-45c3-9fdc-fc00bc12f141" (UID: "9c35fda7-eeab-45c3-9fdc-fc00bc12f141"). InnerVolumeSpecName "kube-api-access-zgfp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:17.867570 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:17.867539    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgfp9\" (UniqueName: \"kubernetes.io/projected/9c35fda7-eeab-45c3-9fdc-fc00bc12f141-kube-api-access-zgfp9\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:18.028198 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:18.028168    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"]
Oct 30 18:55:18.033086 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:18.033060    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f2ceb5-849887dcdc-ddf99"]
Oct 30 18:55:18.203748 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:18.203708    4372 patch_prober.go:29] interesting pod/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:55:18.203948 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:18.203770    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh" podUID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 18:55:18.266425 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:18.266357    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d5c2659-c280-40fb-b2da-bf7672b55833" path="/var/lib/kubelet/pods/1d5c2659-c280-40fb-b2da-bf7672b55833/volumes"
Oct 30 18:55:18.266752 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:18.266738    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e51e05-f986-45b9-836d-6339a685318d" path="/var/lib/kubelet/pods/55e51e05-f986-45b9-836d-6339a685318d/volumes"
Oct 30 18:55:18.267116 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:18.267103    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c35fda7-eeab-45c3-9fdc-fc00bc12f141" path="/var/lib/kubelet/pods/9c35fda7-eeab-45c3-9fdc-fc00bc12f141/volumes"
Oct 30 18:55:19.198153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:19.197218    4372 patch_prober.go:29] interesting pod/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:55:19.198153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:19.197272    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh" podUID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 18:55:20.203499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:20.201229    4372 patch_prober.go:29] interesting pod/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 18:55:20.203499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:20.201306    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh" podUID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 18:55:21.198046 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:21.198014    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:55:21.198439 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:21.198416    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:55:21.199117 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:21.199099    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:55:21.322388 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:21.322356    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"]
Oct 30 18:55:21.323214 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:21.323181    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk" podUID="15bc1bcf-0466-40c4-98af-928b2f3a259d" containerName="istio-proxy" containerID="cri-o://29a271336ed2a33008b7c205aa861aa987c13614e640ddc795047d35b988f94b" gracePeriod=30
Oct 30 18:55:22.358634 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:22.358581    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:55:23.098805 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.098775    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"]
Oct 30 18:55:23.099009 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.098984    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng" podUID="16453557-85f1-483b-8439-449950e8852d" containerName="maas-api" containerID="cri-o://11cb5d10cf09b626ba80f9424ef09e3e0171964d047eee5aea04e05c4ede5f54" gracePeriod=30
Oct 30 18:55:23.548470 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.547044    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:55:23.610455 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.610420    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-metrics-certs\") pod \"16453557-85f1-483b-8439-449950e8852d\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") "
Oct 30 18:55:23.610641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.610543    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-maas-api-tls\") pod \"16453557-85f1-483b-8439-449950e8852d\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") "
Oct 30 18:55:23.610719 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.610707    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16453557-85f1-483b-8439-449950e8852d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16453557-85f1-483b-8439-449950e8852d-openshift-service-ca\") pod \"16453557-85f1-483b-8439-449950e8852d\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") "
Oct 30 18:55:23.610843 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.610815    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16453557-85f1-483b-8439-449950e8852d-kube-api-access-h42tq\" (UniqueName: \"kubernetes.io/projected/16453557-85f1-483b-8439-449950e8852d-kube-api-access-h42tq\") pod \"16453557-85f1-483b-8439-449950e8852d\" (UID: \"16453557-85f1-483b-8439-449950e8852d\") "
Oct 30 18:55:23.611314 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.611157    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16453557-85f1-483b-8439-449950e8852d-openshift-service-ca" pod "16453557-85f1-483b-8439-449950e8852d" (UID: "16453557-85f1-483b-8439-449950e8852d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:23.611544 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.611517    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16453557-85f1-483b-8439-449950e8852d-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:23.613394 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.613349    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-metrics-certs" pod "16453557-85f1-483b-8439-449950e8852d" (UID: "16453557-85f1-483b-8439-449950e8852d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:23.614824 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.614798    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16453557-85f1-483b-8439-449950e8852d-kube-api-access-h42tq" pod "16453557-85f1-483b-8439-449950e8852d" (UID: "16453557-85f1-483b-8439-449950e8852d"). InnerVolumeSpecName "kube-api-access-h42tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:23.614954 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.614933    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-maas-api-tls" pod "16453557-85f1-483b-8439-449950e8852d" (UID: "16453557-85f1-483b-8439-449950e8852d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:23.712606 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.712584    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h42tq\" (UniqueName: \"kubernetes.io/projected/16453557-85f1-483b-8439-449950e8852d-kube-api-access-h42tq\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:23.712702 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.712608    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:23.712702 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.712619    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16453557-85f1-483b-8439-449950e8852d-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:23.742077 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.742052    4372 generic.go:356] "Generic (PLEG): container finished" podID="16453557-85f1-483b-8439-449950e8852d" containerID="11cb5d10cf09b626ba80f9424ef09e3e0171964d047eee5aea04e05c4ede5f54" exitCode=0
Oct 30 18:55:23.742196 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.742099    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"
Oct 30 18:55:23.742196 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.742153    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng" event={"ID":"16453557-85f1-483b-8439-449950e8852d","Type":"ContainerDied","Data":"11cb5d10cf09b626ba80f9424ef09e3e0171964d047eee5aea04e05c4ede5f54"}
Oct 30 18:55:23.742294 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.742208    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng" event={"ID":"16453557-85f1-483b-8439-449950e8852d","Type":"ContainerDied","Data":"f3943aa0157bd750ee92d9f067ea1d0672e1362224d63ec87282d8a2eb4662ce"}
Oct 30 18:55:23.742294 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.742231    4372 scope.go:122] "RemoveContainer" containerID="11cb5d10cf09b626ba80f9424ef09e3e0171964d047eee5aea04e05c4ede5f54"
Oct 30 18:55:23.752600 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.752574    4372 scope.go:122] "RemoveContainer" containerID="11cb5d10cf09b626ba80f9424ef09e3e0171964d047eee5aea04e05c4ede5f54"
Oct 30 18:55:23.752917 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:55:23.752898    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11cb5d10cf09b626ba80f9424ef09e3e0171964d047eee5aea04e05c4ede5f54\": container with ID starting with 11cb5d10cf09b626ba80f9424ef09e3e0171964d047eee5aea04e05c4ede5f54 not found: ID does not exist" containerID="11cb5d10cf09b626ba80f9424ef09e3e0171964d047eee5aea04e05c4ede5f54"
Oct 30 18:55:23.752979 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.752928    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cb5d10cf09b626ba80f9424ef09e3e0171964d047eee5aea04e05c4ede5f54"} err="failed to get container status \"11cb5d10cf09b626ba80f9424ef09e3e0171964d047eee5aea04e05c4ede5f54\": rpc error: code = NotFound desc = could not find container \"11cb5d10cf09b626ba80f9424ef09e3e0171964d047eee5aea04e05c4ede5f54\": container with ID starting with 11cb5d10cf09b626ba80f9424ef09e3e0171964d047eee5aea04e05c4ede5f54 not found: ID does not exist"
Oct 30 18:55:23.856246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.852346    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"]
Oct 30 18:55:23.867149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:23.862917    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2ceb5-5f775f87fd-bgnng"]
Oct 30 18:55:24.266448 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:24.266376    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16453557-85f1-483b-8439-449950e8852d" path="/var/lib/kubelet/pods/16453557-85f1-483b-8439-449950e8852d/volumes"
Oct 30 18:55:25.076430 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:25.076031    4372 patch_prober.go:29] interesting pod/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 18:55:25.076430 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:25.076156    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk" podUID="15bc1bcf-0466-40c4-98af-928b2f3a259d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:55:26.632236 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.632211    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:55:26.764528 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.764494    4372 generic.go:356] "Generic (PLEG): container finished" podID="15bc1bcf-0466-40c4-98af-928b2f3a259d" containerID="29a271336ed2a33008b7c205aa861aa987c13614e640ddc795047d35b988f94b" exitCode=0
Oct 30 18:55:26.764681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.764545    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"
Oct 30 18:55:26.764681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.764544    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk" event={"ID":"15bc1bcf-0466-40c4-98af-928b2f3a259d","Type":"ContainerDied","Data":"29a271336ed2a33008b7c205aa861aa987c13614e640ddc795047d35b988f94b"}
Oct 30 18:55:26.764681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.764664    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk" event={"ID":"15bc1bcf-0466-40c4-98af-928b2f3a259d","Type":"ContainerDied","Data":"9f8b2de42e70cbb1efb95783cf8ca866f7bbfb023c29f8fff26d31dfe3dc5e2e"}
Oct 30 18:55:26.764786 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.764692    4372 scope.go:122] "RemoveContainer" containerID="29a271336ed2a33008b7c205aa861aa987c13614e640ddc795047d35b988f94b"
Oct 30 18:55:26.775393 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.775366    4372 scope.go:122] "RemoveContainer" containerID="77ef934c9b05baf0b75ac32cc792f9209ad4805e53e7007016ece1e9d6f25808"
Oct 30 18:55:26.785463 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.785439    4372 scope.go:122] "RemoveContainer" containerID="29a271336ed2a33008b7c205aa861aa987c13614e640ddc795047d35b988f94b"
Oct 30 18:55:26.785735 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:55:26.785713    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a271336ed2a33008b7c205aa861aa987c13614e640ddc795047d35b988f94b\": container with ID starting with 29a271336ed2a33008b7c205aa861aa987c13614e640ddc795047d35b988f94b not found: ID does not exist" containerID="29a271336ed2a33008b7c205aa861aa987c13614e640ddc795047d35b988f94b"
Oct 30 18:55:26.785794 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.785746    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a271336ed2a33008b7c205aa861aa987c13614e640ddc795047d35b988f94b"} err="failed to get container status \"29a271336ed2a33008b7c205aa861aa987c13614e640ddc795047d35b988f94b\": rpc error: code = NotFound desc = could not find container \"29a271336ed2a33008b7c205aa861aa987c13614e640ddc795047d35b988f94b\": container with ID starting with 29a271336ed2a33008b7c205aa861aa987c13614e640ddc795047d35b988f94b not found: ID does not exist"
Oct 30 18:55:26.785794 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.785763    4372 scope.go:122] "RemoveContainer" containerID="77ef934c9b05baf0b75ac32cc792f9209ad4805e53e7007016ece1e9d6f25808"
Oct 30 18:55:26.786036 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:55:26.786022    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ef934c9b05baf0b75ac32cc792f9209ad4805e53e7007016ece1e9d6f25808\": container with ID starting with 77ef934c9b05baf0b75ac32cc792f9209ad4805e53e7007016ece1e9d6f25808 not found: ID does not exist" containerID="77ef934c9b05baf0b75ac32cc792f9209ad4805e53e7007016ece1e9d6f25808"
Oct 30 18:55:26.786072 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.786040    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ef934c9b05baf0b75ac32cc792f9209ad4805e53e7007016ece1e9d6f25808"} err="failed to get container status \"77ef934c9b05baf0b75ac32cc792f9209ad4805e53e7007016ece1e9d6f25808\": rpc error: code = NotFound desc = could not find container \"77ef934c9b05baf0b75ac32cc792f9209ad4805e53e7007016ece1e9d6f25808\": container with ID starting with 77ef934c9b05baf0b75ac32cc792f9209ad4805e53e7007016ece1e9d6f25808 not found: ID does not exist"
Oct 30 18:55:26.786403 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.786386    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-workload-certs\") pod \"15bc1bcf-0466-40c4-98af-928b2f3a259d\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") "
Oct 30 18:55:26.786449 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.786420    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-data\") pod \"15bc1bcf-0466-40c4-98af-928b2f3a259d\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") "
Oct 30 18:55:26.786485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.786467    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-envoy\") pod \"15bc1bcf-0466-40c4-98af-928b2f3a259d\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") "
Oct 30 18:55:26.786555 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.786540    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-workload-socket\") pod \"15bc1bcf-0466-40c4-98af-928b2f3a259d\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") "
Oct 30 18:55:26.786686 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.786666    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-workload-certs" pod "15bc1bcf-0466-40c4-98af-928b2f3a259d" (UID: "15bc1bcf-0466-40c4-98af-928b2f3a259d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:26.786798 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.786772    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-data" pod "15bc1bcf-0466-40c4-98af-928b2f3a259d" (UID: "15bc1bcf-0466-40c4-98af-928b2f3a259d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:26.786926 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.786905    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-workload-socket" pod "15bc1bcf-0466-40c4-98af-928b2f3a259d" (UID: "15bc1bcf-0466-40c4-98af-928b2f3a259d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:26.786981 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.786929    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-token\" (UniqueName: \"kubernetes.io/projected/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-token\") pod \"15bc1bcf-0466-40c4-98af-928b2f3a259d\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") "
Oct 30 18:55:26.786981 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.786956    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15bc1bcf-0466-40c4-98af-928b2f3a259d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15bc1bcf-0466-40c4-98af-928b2f3a259d-istiod-ca-cert\") pod \"15bc1bcf-0466-40c4-98af-928b2f3a259d\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") "
Oct 30 18:55:26.787065 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.787039    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-podinfo\") pod \"15bc1bcf-0466-40c4-98af-928b2f3a259d\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") "
Oct 30 18:55:26.787119 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.787101    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15bc1bcf-0466-40c4-98af-928b2f3a259d-kube-api-access-rtpwd\" (UniqueName: \"kubernetes.io/projected/15bc1bcf-0466-40c4-98af-928b2f3a259d-kube-api-access-rtpwd\") pod \"15bc1bcf-0466-40c4-98af-928b2f3a259d\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") "
Oct 30 18:55:26.787211 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.787156    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-credential-socket\") pod \"15bc1bcf-0466-40c4-98af-928b2f3a259d\" (UID: \"15bc1bcf-0466-40c4-98af-928b2f3a259d\") "
Oct 30 18:55:26.787417 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.787392    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15bc1bcf-0466-40c4-98af-928b2f3a259d-istiod-ca-cert" pod "15bc1bcf-0466-40c4-98af-928b2f3a259d" (UID: "15bc1bcf-0466-40c4-98af-928b2f3a259d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:26.787974 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.787562    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-credential-socket" pod "15bc1bcf-0466-40c4-98af-928b2f3a259d" (UID: "15bc1bcf-0466-40c4-98af-928b2f3a259d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:26.787974 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.787877    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:26.787974 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.787903    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:26.787974 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.787919    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:26.787974 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.787934    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:26.787974 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.787950    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15bc1bcf-0466-40c4-98af-928b2f3a259d-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:26.789371 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.789349    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-envoy" pod "15bc1bcf-0466-40c4-98af-928b2f3a259d" (UID: "15bc1bcf-0466-40c4-98af-928b2f3a259d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:26.789487 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.789432    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-podinfo" pod "15bc1bcf-0466-40c4-98af-928b2f3a259d" (UID: "15bc1bcf-0466-40c4-98af-928b2f3a259d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:55:26.789487 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.789471    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-token" pod "15bc1bcf-0466-40c4-98af-928b2f3a259d" (UID: "15bc1bcf-0466-40c4-98af-928b2f3a259d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:26.789700 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.789682    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15bc1bcf-0466-40c4-98af-928b2f3a259d-kube-api-access-rtpwd" pod "15bc1bcf-0466-40c4-98af-928b2f3a259d" (UID: "15bc1bcf-0466-40c4-98af-928b2f3a259d"). InnerVolumeSpecName "kube-api-access-rtpwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:26.889734 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.889653    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:26.889734 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.889683    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:26.889734 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.889693    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15bc1bcf-0466-40c4-98af-928b2f3a259d-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:26.889734 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:26.889703    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtpwd\" (UniqueName: \"kubernetes.io/projected/15bc1bcf-0466-40c4-98af-928b2f3a259d-kube-api-access-rtpwd\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:27.091086 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:27.091046    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"]
Oct 30 18:55:27.095212 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:27.095188    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d5ab51-openshift-default-789876c757-zl8gk"]
Oct 30 18:55:27.700024 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:55:27.699981    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:55:28.267652 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:28.267623    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15bc1bcf-0466-40c4-98af-928b2f3a259d" path="/var/lib/kubelet/pods/15bc1bcf-0466-40c4-98af-928b2f3a259d/volumes"
Oct 30 18:55:33.712988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:33.712952    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"]
Oct 30 18:55:33.713489 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:33.713169    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2" podUID="825a5ded-653f-4172-9001-472d28b62d02" containerName="maas-api" containerID="cri-o://b4494e7ad13c1ca692b4d6567882af680a81fadb78264f8de3d0ea7224961470" gracePeriod=30
Oct 30 18:55:33.784075 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:55:33.784027    4372 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825a5ded_653f_4172_9001_472d28b62d02.slice/crio-b4494e7ad13c1ca692b4d6567882af680a81fadb78264f8de3d0ea7224961470.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825a5ded_653f_4172_9001_472d28b62d02.slice/crio-conmon-b4494e7ad13c1ca692b4d6567882af680a81fadb78264f8de3d0ea7224961470.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 18:55:33.847074 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:33.847043    4372 generic.go:356] "Generic (PLEG): container finished" podID="825a5ded-653f-4172-9001-472d28b62d02" containerID="b4494e7ad13c1ca692b4d6567882af680a81fadb78264f8de3d0ea7224961470" exitCode=0
Oct 30 18:55:33.847235 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:33.847157    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2" event={"ID":"825a5ded-653f-4172-9001-472d28b62d02","Type":"ContainerDied","Data":"b4494e7ad13c1ca692b4d6567882af680a81fadb78264f8de3d0ea7224961470"}
Oct 30 18:55:34.148326 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.146976    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:55:34.265666 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.265632    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/825a5ded-653f-4172-9001-472d28b62d02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825a5ded-653f-4172-9001-472d28b62d02-maas-api-tls\") pod \"825a5ded-653f-4172-9001-472d28b62d02\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") "
Oct 30 18:55:34.265820 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.265687    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825a5ded-653f-4172-9001-472d28b62d02-kube-api-access-qzrhk\" (UniqueName: \"kubernetes.io/projected/825a5ded-653f-4172-9001-472d28b62d02-kube-api-access-qzrhk\") pod \"825a5ded-653f-4172-9001-472d28b62d02\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") "
Oct 30 18:55:34.265906 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.265880    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/825a5ded-653f-4172-9001-472d28b62d02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/825a5ded-653f-4172-9001-472d28b62d02-openshift-service-ca\") pod \"825a5ded-653f-4172-9001-472d28b62d02\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") "
Oct 30 18:55:34.265971 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.265926    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/825a5ded-653f-4172-9001-472d28b62d02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a5ded-653f-4172-9001-472d28b62d02-metrics-certs\") pod \"825a5ded-653f-4172-9001-472d28b62d02\" (UID: \"825a5ded-653f-4172-9001-472d28b62d02\") "
Oct 30 18:55:34.266345 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.266315    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825a5ded-653f-4172-9001-472d28b62d02-openshift-service-ca" pod "825a5ded-653f-4172-9001-472d28b62d02" (UID: "825a5ded-653f-4172-9001-472d28b62d02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:34.266525 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.266495    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/825a5ded-653f-4172-9001-472d28b62d02-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:34.267957 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.267927    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825a5ded-653f-4172-9001-472d28b62d02-kube-api-access-qzrhk" pod "825a5ded-653f-4172-9001-472d28b62d02" (UID: "825a5ded-653f-4172-9001-472d28b62d02"). InnerVolumeSpecName "kube-api-access-qzrhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:34.268067 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.267958    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825a5ded-653f-4172-9001-472d28b62d02-maas-api-tls" pod "825a5ded-653f-4172-9001-472d28b62d02" (UID: "825a5ded-653f-4172-9001-472d28b62d02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:34.268370 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.268353    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825a5ded-653f-4172-9001-472d28b62d02-metrics-certs" pod "825a5ded-653f-4172-9001-472d28b62d02" (UID: "825a5ded-653f-4172-9001-472d28b62d02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:34.369651 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.369607    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a5ded-653f-4172-9001-472d28b62d02-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:34.369651 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.369653    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825a5ded-653f-4172-9001-472d28b62d02-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:34.369938 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.369670    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzrhk\" (UniqueName: \"kubernetes.io/projected/825a5ded-653f-4172-9001-472d28b62d02-kube-api-access-qzrhk\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:34.855550 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.855513    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"
Oct 30 18:55:34.855980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.855553    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2" event={"ID":"825a5ded-653f-4172-9001-472d28b62d02","Type":"ContainerDied","Data":"070e792b42699463c1feb069e768a4689d2a28453127222b868144265a5a3bf4"}
Oct 30 18:55:34.855980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.855609    4372 scope.go:122] "RemoveContainer" containerID="b4494e7ad13c1ca692b4d6567882af680a81fadb78264f8de3d0ea7224961470"
Oct 30 18:55:34.901217 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.901191    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"]
Oct 30 18:55:34.907301 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:34.907280    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9d42ff-874f78fc5-zt5k2"]
Oct 30 18:55:36.265736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:36.265702    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="825a5ded-653f-4172-9001-472d28b62d02" path="/var/lib/kubelet/pods/825a5ded-653f-4172-9001-472d28b62d02/volumes"
Oct 30 18:55:41.723678 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:41.723644    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"]
Oct 30 18:55:41.734276 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:41.734242    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"]
Oct 30 18:55:41.862711 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:41.862674    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"]
Oct 30 18:55:41.862980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:41.862952    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh" podUID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" containerName="istio-proxy" containerID="cri-o://908f1dbf6602584fd0ded7e1e1a45049b2c2f5664c3e2c7f1d193cad6d0603f2" gracePeriod=30
Oct 30 18:55:44.174402 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.174369    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"]
Oct 30 18:55:44.174904 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.174642    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97" podUID="06507e5a-e223-450e-8fb8-30b8d059aa04" containerName="maas-api" containerID="cri-o://e7e4d048082885baa435d7e6a214972c444e311b24919acbdf4b10c0c229a5b6" gracePeriod=30
Oct 30 18:55:44.507222 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.495873    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:55:44.575358 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.573583    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06507e5a-e223-450e-8fb8-30b8d059aa04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06507e5a-e223-450e-8fb8-30b8d059aa04-metrics-certs\") pod \"06507e5a-e223-450e-8fb8-30b8d059aa04\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") "
Oct 30 18:55:44.575358 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.573678    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06507e5a-e223-450e-8fb8-30b8d059aa04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06507e5a-e223-450e-8fb8-30b8d059aa04-openshift-service-ca\") pod \"06507e5a-e223-450e-8fb8-30b8d059aa04\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") "
Oct 30 18:55:44.575358 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.573760    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06507e5a-e223-450e-8fb8-30b8d059aa04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06507e5a-e223-450e-8fb8-30b8d059aa04-maas-api-tls\") pod \"06507e5a-e223-450e-8fb8-30b8d059aa04\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") "
Oct 30 18:55:44.575358 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.573826    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06507e5a-e223-450e-8fb8-30b8d059aa04-kube-api-access-6bmc4\" (UniqueName: \"kubernetes.io/projected/06507e5a-e223-450e-8fb8-30b8d059aa04-kube-api-access-6bmc4\") pod \"06507e5a-e223-450e-8fb8-30b8d059aa04\" (UID: \"06507e5a-e223-450e-8fb8-30b8d059aa04\") "
Oct 30 18:55:44.577741 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.577442    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06507e5a-e223-450e-8fb8-30b8d059aa04-openshift-service-ca" pod "06507e5a-e223-450e-8fb8-30b8d059aa04" (UID: "06507e5a-e223-450e-8fb8-30b8d059aa04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:44.595713 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.595659    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06507e5a-e223-450e-8fb8-30b8d059aa04-metrics-certs" pod "06507e5a-e223-450e-8fb8-30b8d059aa04" (UID: "06507e5a-e223-450e-8fb8-30b8d059aa04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:44.602174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.596635    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06507e5a-e223-450e-8fb8-30b8d059aa04-maas-api-tls" pod "06507e5a-e223-450e-8fb8-30b8d059aa04" (UID: "06507e5a-e223-450e-8fb8-30b8d059aa04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:44.608731 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.608697    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06507e5a-e223-450e-8fb8-30b8d059aa04-kube-api-access-6bmc4" pod "06507e5a-e223-450e-8fb8-30b8d059aa04" (UID: "06507e5a-e223-450e-8fb8-30b8d059aa04"). InnerVolumeSpecName "kube-api-access-6bmc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:44.675584 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.675543    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06507e5a-e223-450e-8fb8-30b8d059aa04-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:44.675584 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.675576    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06507e5a-e223-450e-8fb8-30b8d059aa04-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:44.675584 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.675590    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bmc4\" (UniqueName: \"kubernetes.io/projected/06507e5a-e223-450e-8fb8-30b8d059aa04-kube-api-access-6bmc4\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:44.675816 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.675604    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06507e5a-e223-450e-8fb8-30b8d059aa04-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:44.962555 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.962516    4372 generic.go:356] "Generic (PLEG): container finished" podID="06507e5a-e223-450e-8fb8-30b8d059aa04" containerID="e7e4d048082885baa435d7e6a214972c444e311b24919acbdf4b10c0c229a5b6" exitCode=0
Oct 30 18:55:44.962747 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.962561    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"
Oct 30 18:55:44.962747 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.962561    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97" event={"ID":"06507e5a-e223-450e-8fb8-30b8d059aa04","Type":"ContainerDied","Data":"e7e4d048082885baa435d7e6a214972c444e311b24919acbdf4b10c0c229a5b6"}
Oct 30 18:55:44.962747 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.962702    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97" event={"ID":"06507e5a-e223-450e-8fb8-30b8d059aa04","Type":"ContainerDied","Data":"59b643b62f68bbf6631c6aa7b072910cb47b404b1d71ec54b82ea9f4a0bfd083"}
Oct 30 18:55:44.962747 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.962745    4372 scope.go:122] "RemoveContainer" containerID="e7e4d048082885baa435d7e6a214972c444e311b24919acbdf4b10c0c229a5b6"
Oct 30 18:55:44.977262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.977237    4372 scope.go:122] "RemoveContainer" containerID="e7e4d048082885baa435d7e6a214972c444e311b24919acbdf4b10c0c229a5b6"
Oct 30 18:55:44.977799 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:55:44.977778    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e4d048082885baa435d7e6a214972c444e311b24919acbdf4b10c0c229a5b6\": container with ID starting with e7e4d048082885baa435d7e6a214972c444e311b24919acbdf4b10c0c229a5b6 not found: ID does not exist" containerID="e7e4d048082885baa435d7e6a214972c444e311b24919acbdf4b10c0c229a5b6"
Oct 30 18:55:44.977900 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.977810    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e4d048082885baa435d7e6a214972c444e311b24919acbdf4b10c0c229a5b6"} err="failed to get container status \"e7e4d048082885baa435d7e6a214972c444e311b24919acbdf4b10c0c229a5b6\": rpc error: code = NotFound desc = could not find container \"e7e4d048082885baa435d7e6a214972c444e311b24919acbdf4b10c0c229a5b6\": container with ID starting with e7e4d048082885baa435d7e6a214972c444e311b24919acbdf4b10c0c229a5b6 not found: ID does not exist"
Oct 30 18:55:44.991672 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.991651    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"]
Oct 30 18:55:44.996661 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:44.996641    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-59f140-75f8b54f8c-tqh97"]
Oct 30 18:55:46.265488 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:46.265461    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06507e5a-e223-450e-8fb8-30b8d059aa04" path="/var/lib/kubelet/pods/06507e5a-e223-450e-8fb8-30b8d059aa04/volumes"
Oct 30 18:55:46.724342 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:46.724280    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl" podUID="a513a47e-5603-4b6c-a1e1-48bcd0c85a32" containerName="payload-processing" containerID="cri-o://9b9fa464ea87f95cffe063e91b0bf6102489a7632ac45abf6e120016df681461" gracePeriod=25
Oct 30 18:55:46.735076 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:46.735046    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m" podUID="8a0503b0-52db-42f1-9904-fa6395aeb865" containerName="payload-pre-processing" containerID="cri-o://a7bd6ebc484f7735b416178c341f9bbbd94716f1c2ea62ca910d964f9b8f1f42" gracePeriod=25
Oct 30 18:55:46.983078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:46.983018    4372 generic.go:356] "Generic (PLEG): container finished" podID="8a0503b0-52db-42f1-9904-fa6395aeb865" containerID="a7bd6ebc484f7735b416178c341f9bbbd94716f1c2ea62ca910d964f9b8f1f42" exitCode=0
Oct 30 18:55:46.983294 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:46.983266    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m" event={"ID":"8a0503b0-52db-42f1-9904-fa6395aeb865","Type":"ContainerDied","Data":"a7bd6ebc484f7735b416178c341f9bbbd94716f1c2ea62ca910d964f9b8f1f42"}
Oct 30 18:55:46.989331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:46.989306    4372 generic.go:356] "Generic (PLEG): container finished" podID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" containerID="908f1dbf6602584fd0ded7e1e1a45049b2c2f5664c3e2c7f1d193cad6d0603f2" exitCode=0
Oct 30 18:55:46.989495 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:46.989385    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh" event={"ID":"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4","Type":"ContainerDied","Data":"908f1dbf6602584fd0ded7e1e1a45049b2c2f5664c3e2c7f1d193cad6d0603f2"}
Oct 30 18:55:46.989495 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:46.989426    4372 scope.go:122] "RemoveContainer" containerID="531c3282d0ffad66fa9b62638c247e4d7124c4b1ecb08c3438e4c5936fabef36"
Oct 30 18:55:46.992600 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:46.992579    4372 generic.go:356] "Generic (PLEG): container finished" podID="a513a47e-5603-4b6c-a1e1-48bcd0c85a32" containerID="9b9fa464ea87f95cffe063e91b0bf6102489a7632ac45abf6e120016df681461" exitCode=0
Oct 30 18:55:46.992720 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:46.992666    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl" event={"ID":"a513a47e-5603-4b6c-a1e1-48bcd0c85a32","Type":"ContainerDied","Data":"9b9fa464ea87f95cffe063e91b0bf6102489a7632ac45abf6e120016df681461"}
Oct 30 18:55:47.027327 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.027301    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"
Oct 30 18:55:47.036085 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.036059    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"
Oct 30 18:55:47.136942 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.136909    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a0503b0-52db-42f1-9904-fa6395aeb865-kube-api-access-rls4d\" (UniqueName: \"kubernetes.io/projected/8a0503b0-52db-42f1-9904-fa6395aeb865-kube-api-access-rls4d\") pod \"8a0503b0-52db-42f1-9904-fa6395aeb865\" (UID: \"8a0503b0-52db-42f1-9904-fa6395aeb865\") "
Oct 30 18:55:47.137147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.137110    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a513a47e-5603-4b6c-a1e1-48bcd0c85a32-kube-api-access-9z8qn\" (UniqueName: \"kubernetes.io/projected/a513a47e-5603-4b6c-a1e1-48bcd0c85a32-kube-api-access-9z8qn\") pod \"a513a47e-5603-4b6c-a1e1-48bcd0c85a32\" (UID: \"a513a47e-5603-4b6c-a1e1-48bcd0c85a32\") "
Oct 30 18:55:47.137577 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.137545    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a0503b0-52db-42f1-9904-fa6395aeb865-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a0503b0-52db-42f1-9904-fa6395aeb865-plugins-config-volume\") pod \"8a0503b0-52db-42f1-9904-fa6395aeb865\" (UID: \"8a0503b0-52db-42f1-9904-fa6395aeb865\") "
Oct 30 18:55:47.138194 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.138175    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a513a47e-5603-4b6c-a1e1-48bcd0c85a32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a513a47e-5603-4b6c-a1e1-48bcd0c85a32-plugins-config-volume\") pod \"a513a47e-5603-4b6c-a1e1-48bcd0c85a32\" (UID: \"a513a47e-5603-4b6c-a1e1-48bcd0c85a32\") "
Oct 30 18:55:47.138524 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.138051    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0503b0-52db-42f1-9904-fa6395aeb865-plugins-config-volume" pod "8a0503b0-52db-42f1-9904-fa6395aeb865" (UID: "8a0503b0-52db-42f1-9904-fa6395aeb865"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:47.138609 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.138548    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a513a47e-5603-4b6c-a1e1-48bcd0c85a32-plugins-config-volume" pod "a513a47e-5603-4b6c-a1e1-48bcd0c85a32" (UID: "a513a47e-5603-4b6c-a1e1-48bcd0c85a32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:47.139248 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.139224    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a0503b0-52db-42f1-9904-fa6395aeb865-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:47.139369 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.139355    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a513a47e-5603-4b6c-a1e1-48bcd0c85a32-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:47.139985 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.139958    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a513a47e-5603-4b6c-a1e1-48bcd0c85a32-kube-api-access-9z8qn" pod "a513a47e-5603-4b6c-a1e1-48bcd0c85a32" (UID: "a513a47e-5603-4b6c-a1e1-48bcd0c85a32"). InnerVolumeSpecName "kube-api-access-9z8qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:47.140441 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.140421    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0503b0-52db-42f1-9904-fa6395aeb865-kube-api-access-rls4d" pod "8a0503b0-52db-42f1-9904-fa6395aeb865" (UID: "8a0503b0-52db-42f1-9904-fa6395aeb865"). InnerVolumeSpecName "kube-api-access-rls4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:47.246204 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.243651    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rls4d\" (UniqueName: \"kubernetes.io/projected/8a0503b0-52db-42f1-9904-fa6395aeb865-kube-api-access-rls4d\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:47.246204 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.243690    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z8qn\" (UniqueName: \"kubernetes.io/projected/a513a47e-5603-4b6c-a1e1-48bcd0c85a32-kube-api-access-9z8qn\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:47.522189 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.522161    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:55:47.650008 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.649973    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-credential-socket\") pod \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") "
Oct 30 18:55:47.650227 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.650068    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-data\") pod \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") "
Oct 30 18:55:47.650227 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.650150    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-workload-certs\") pod \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") "
Oct 30 18:55:47.650227 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.650184    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-envoy\") pod \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") "
Oct 30 18:55:47.650227 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.650215    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istiod-ca-cert\") pod \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") "
Oct 30 18:55:47.650485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.650268    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-credential-socket" pod "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" (UID: "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:47.650485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.650308    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-podinfo\") pod \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") "
Oct 30 18:55:47.650485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.650337    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-workload-socket\") pod \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") "
Oct 30 18:55:47.650485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.650418    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-kube-api-access-xwk9z\" (UniqueName: \"kubernetes.io/projected/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-kube-api-access-xwk9z\") pod \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") "
Oct 30 18:55:47.650736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.650517    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-token\" (UniqueName: \"kubernetes.io/projected/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-token\") pod \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\" (UID: \"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4\") "
Oct 30 18:55:47.650736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.650535    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-workload-certs" pod "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" (UID: "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:47.650736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.650663    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-data" pod "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" (UID: "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:47.651246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.651119    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istiod-ca-cert" pod "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" (UID: "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:47.651246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.651148    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:47.651246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.651195    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:47.651246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.651209    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:47.651614 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.651437    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-workload-socket" pod "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" (UID: "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:47.652949 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.652925    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-token" pod "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" (UID: "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:47.653300 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.653278    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-podinfo" pod "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" (UID: "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:55:47.653300 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.653280    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-kube-api-access-xwk9z" pod "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" (UID: "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4"). InnerVolumeSpecName "kube-api-access-xwk9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:47.653551 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.653531    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-envoy" pod "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" (UID: "2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:47.752645 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.752618    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:47.752645 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.752645    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:47.752808 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.752660    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:47.752808 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.752673    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:47.752808 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.752686    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:47.752808 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:47.752700    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwk9z\" (UniqueName: \"kubernetes.io/projected/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4-kube-api-access-xwk9z\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:48.000659 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.000632    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"
Oct 30 18:55:48.000659 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.000636    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m" event={"ID":"8a0503b0-52db-42f1-9904-fa6395aeb865","Type":"ContainerDied","Data":"f03461a87c5041e30de69af8a2756acfcb308b72218bbae115e60940ef82a0ef"}
Oct 30 18:55:48.000856 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.000683    4372 scope.go:122] "RemoveContainer" containerID="a7bd6ebc484f7735b416178c341f9bbbd94716f1c2ea62ca910d964f9b8f1f42"
Oct 30 18:55:48.002354 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.002335    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"
Oct 30 18:55:48.002354 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.002346    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh" event={"ID":"2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4","Type":"ContainerDied","Data":"6a0dec251b5363fd9708d1e37278749b02b10605e6772fdab5d33aa49a1de5c9"}
Oct 30 18:55:48.004049 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.004020    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"
Oct 30 18:55:48.004177 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.004049    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl" event={"ID":"a513a47e-5603-4b6c-a1e1-48bcd0c85a32","Type":"ContainerDied","Data":"8aafc8714cc0f922bdfd798d7973aebc3027a238f56f4f4d3904f3076ace6cc0"}
Oct 30 18:55:48.020660 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.020599    4372 scope.go:122] "RemoveContainer" containerID="908f1dbf6602584fd0ded7e1e1a45049b2c2f5664c3e2c7f1d193cad6d0603f2"
Oct 30 18:55:48.029833 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.029820    4372 scope.go:122] "RemoveContainer" containerID="9b9fa464ea87f95cffe063e91b0bf6102489a7632ac45abf6e120016df681461"
Oct 30 18:55:48.041326 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.041306    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"]
Oct 30 18:55:48.055308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.055285    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-59f140-bb9c987bc-nsz6m"]
Oct 30 18:55:48.075097 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.075077    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"]
Oct 30 18:55:48.081348 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.081331    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-59f140-6d5769f5b8-726bl"]
Oct 30 18:55:48.098468 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.098445    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"]
Oct 30 18:55:48.103417 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.103397    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9d42ff-openshift-default-6d59484b8f-c2khh"]
Oct 30 18:55:48.266031 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.265973    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" path="/var/lib/kubelet/pods/2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4/volumes"
Oct 30 18:55:48.266428 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.266413    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a0503b0-52db-42f1-9904-fa6395aeb865" path="/var/lib/kubelet/pods/8a0503b0-52db-42f1-9904-fa6395aeb865/volumes"
Oct 30 18:55:48.266808 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:48.266795    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a513a47e-5603-4b6c-a1e1-48bcd0c85a32" path="/var/lib/kubelet/pods/a513a47e-5603-4b6c-a1e1-48bcd0c85a32/volumes"
Oct 30 18:55:51.763068 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:51.763040    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg_4fa388c0-7e72-4b86-b692-53d9c53ac30b/payload-processing/0.log"
Oct 30 18:55:51.945802 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:55:51.945781    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg_4fa388c0-7e72-4b86-b692-53d9c53ac30b/payload-processing/0.log"
Oct 30 18:55:57.696348 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:55:57.696307    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:56:01.044342 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.044306    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"]
Oct 30 18:56:01.045049 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045032    4372 cpu_manager.go:414] "containerMap: removing container" podUID="06507e5a-e223-450e-8fb8-30b8d059aa04" containerName="maas-api"
Oct 30 18:56:01.045110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045052    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06507e5a-e223-450e-8fb8-30b8d059aa04" containerName="maas-api"
Oct 30 18:56:01.045110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045068    4372 cpu_manager.go:414] "containerMap: removing container" podUID="1d5c2659-c280-40fb-b2da-bf7672b55833" containerName="payload-processing"
Oct 30 18:56:01.045110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045073    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d5c2659-c280-40fb-b2da-bf7672b55833" containerName="payload-processing"
Oct 30 18:56:01.045110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045083    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f6846d27-0053-4c6b-810a-1eab95f42638" containerName="maas-api"
Oct 30 18:56:01.045110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045088    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6846d27-0053-4c6b-810a-1eab95f42638" containerName="maas-api"
Oct 30 18:56:01.045110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045096    4372 cpu_manager.go:414] "containerMap: removing container" podUID="825a5ded-653f-4172-9001-472d28b62d02" containerName="maas-api"
Oct 30 18:56:01.045110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045100    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825a5ded-653f-4172-9001-472d28b62d02" containerName="maas-api"
Oct 30 18:56:01.045110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045108    4372 cpu_manager.go:414] "containerMap: removing container" podUID="55e51e05-f986-45b9-836d-6339a685318d" containerName="payload-processing"
Oct 30 18:56:01.045110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045114    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e51e05-f986-45b9-836d-6339a685318d" containerName="payload-processing"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045138    4372 cpu_manager.go:414] "containerMap: removing container" podUID="a513a47e-5603-4b6c-a1e1-48bcd0c85a32" containerName="payload-processing"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045143    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a513a47e-5603-4b6c-a1e1-48bcd0c85a32" containerName="payload-processing"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045154    4372 cpu_manager.go:414] "containerMap: removing container" podUID="9c35fda7-eeab-45c3-9fdc-fc00bc12f141" containerName="payload-pre-processing"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045159    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c35fda7-eeab-45c3-9fdc-fc00bc12f141" containerName="payload-pre-processing"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045177    4372 cpu_manager.go:414] "containerMap: removing container" podUID="15bc1bcf-0466-40c4-98af-928b2f3a259d" containerName="istio-proxy"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045182    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15bc1bcf-0466-40c4-98af-928b2f3a259d" containerName="istio-proxy"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045193    4372 cpu_manager.go:414] "containerMap: removing container" podUID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" containerName="istio-proxy"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045197    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" containerName="istio-proxy"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045203    4372 cpu_manager.go:414] "containerMap: removing container" podUID="15bc1bcf-0466-40c4-98af-928b2f3a259d" containerName="istio-proxy"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045207    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15bc1bcf-0466-40c4-98af-928b2f3a259d" containerName="istio-proxy"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045218    4372 cpu_manager.go:414] "containerMap: removing container" podUID="16453557-85f1-483b-8439-449950e8852d" containerName="maas-api"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045222    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16453557-85f1-483b-8439-449950e8852d" containerName="maas-api"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045231    4372 cpu_manager.go:414] "containerMap: removing container" podUID="8a0503b0-52db-42f1-9904-fa6395aeb865" containerName="payload-pre-processing"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045235    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a0503b0-52db-42f1-9904-fa6395aeb865" containerName="payload-pre-processing"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045334    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e51e05-f986-45b9-836d-6339a685318d" containerName="payload-processing"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045342    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="a513a47e-5603-4b6c-a1e1-48bcd0c85a32" containerName="payload-processing"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045348    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c35fda7-eeab-45c3-9fdc-fc00bc12f141" containerName="payload-pre-processing"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045356    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="15bc1bcf-0466-40c4-98af-928b2f3a259d" containerName="istio-proxy"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045362    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" containerName="istio-proxy"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045368    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="15bc1bcf-0466-40c4-98af-928b2f3a259d" containerName="istio-proxy"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045373    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" containerName="istio-proxy"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045379    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="16453557-85f1-483b-8439-449950e8852d" containerName="maas-api"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045385    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a0503b0-52db-42f1-9904-fa6395aeb865" containerName="payload-pre-processing"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045392    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="06507e5a-e223-450e-8fb8-30b8d059aa04" containerName="maas-api"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045398    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d5c2659-c280-40fb-b2da-bf7672b55833" containerName="payload-processing"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045404    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6846d27-0053-4c6b-810a-1eab95f42638" containerName="maas-api"
Oct 30 18:56:01.045406 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.045409    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="825a5ded-653f-4172-9001-472d28b62d02" containerName="maas-api"
Oct 30 18:56:01.048848 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.048828    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.054246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.054224    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-50b19d-openshift-default-dockercfg-558t7\""
Oct 30 18:56:01.077526 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.077504    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"]
Oct 30 18:56:01.163151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.153685    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-token\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.163151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.153820    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c448eb1a-dfec-4669-a5e7-585087730fbc-istiod-ca-cert\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.163151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.153871    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-credential-socket\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.163151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.153904    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2n8d\" (UniqueName: \"kubernetes.io/projected/c448eb1a-dfec-4669-a5e7-585087730fbc-kube-api-access-w2n8d\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.163151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.153965    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-data\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.163151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.153996    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-envoy\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.163151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.154097    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-podinfo\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.178152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.177318    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-workload-certs\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.178152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.177399    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-workload-socket\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.280081 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.280032    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2n8d\" (UniqueName: \"kubernetes.io/projected/c448eb1a-dfec-4669-a5e7-585087730fbc-kube-api-access-w2n8d\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.280327 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.280119    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-data\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.280327 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.280266    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-envoy\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.280460 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.280362    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-podinfo\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.280460 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.280448    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-workload-certs\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.280565 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.280483    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-workload-socket\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.280565 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.280552    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-token\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.280685 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.280634    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c448eb1a-dfec-4669-a5e7-585087730fbc-istiod-ca-cert\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.280740 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.280680    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-credential-socket\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.284369 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.281334    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-credential-socket\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.284369 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.281906    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-data\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.284369 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.282502    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-workload-socket\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.284369 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.284350    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-workload-certs\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.293158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.285258    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c448eb1a-dfec-4669-a5e7-585087730fbc-istiod-ca-cert\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.293158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.290441    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-podinfo\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.293404 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.293231    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-envoy\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.301222 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.301094    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2n8d\" (UniqueName: \"kubernetes.io/projected/c448eb1a-dfec-4669-a5e7-585087730fbc-kube-api-access-w2n8d\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.304595 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.304029    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-token\") pod \"e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") " pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.363158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.363105    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:01.665436 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:56:01.665403    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc448eb1a_dfec_4669_a5e7_585087730fbc.slice/crio-6cd8c0f4d3c901f66392f1f022d15fa3c31cb4828d36ee5510fb21fd19149659 WatchSource:0}: Error finding container 6cd8c0f4d3c901f66392f1f022d15fa3c31cb4828d36ee5510fb21fd19149659: Status 404 returned error can't find the container with id 6cd8c0f4d3c901f66392f1f022d15fa3c31cb4828d36ee5510fb21fd19149659
Oct 30 18:56:01.665887 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.665864    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"]
Oct 30 18:56:01.668228 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.668186    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:01.668307 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.668279    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:01.668346 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:01.668323    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:02.265387 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:02.265358    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5" event={"ID":"c448eb1a-dfec-4669-a5e7-585087730fbc","Type":"ContainerStarted","Data":"9c76113f3c511d2323e748ba0a7d742c6bfbc6d9e77738b9d827f7c4e27fc626"}
Oct 30 18:56:02.265780 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:02.265403    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5" event={"ID":"c448eb1a-dfec-4669-a5e7-585087730fbc","Type":"ContainerStarted","Data":"6cd8c0f4d3c901f66392f1f022d15fa3c31cb4828d36ee5510fb21fd19149659"}
Oct 30 18:56:02.288255 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:02.288206    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5" podStartSLOduration=1.28817011 podStartE2EDuration="1.28817011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:02.286737885 +0000 UTC m=+1376.155156215" watchObservedRunningTime="2026-10-30 18:56:02.28817011 +0000 UTC m=+1376.156588513"
Oct 30 18:56:02.363504 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:02.363474    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:02.369246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:02.369224    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:03.268632 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:03.268601    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:03.269540 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:03.269518    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:06.198586 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.198552    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"]
Oct 30 18:56:06.199351 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.199331    4372 cpu_manager.go:414] "containerMap: removing container" podUID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" containerName="istio-proxy"
Oct 30 18:56:06.199457 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.199353    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2045c19b-ea71-4ba2-9b6f-fdfe9280e2f4" containerName="istio-proxy"
Oct 30 18:56:06.202968 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.202947    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"
Oct 30 18:56:06.216669 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.216650    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-50b19d\""
Oct 30 18:56:06.230518 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.230498    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"]
Oct 30 18:56:06.242874 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.242850    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"]
Oct 30 18:56:06.246898 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.246881    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"
Oct 30 18:56:06.256180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.256161    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-50b19d-dockercfg-wmw6x\""
Oct 30 18:56:06.276296 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.276273    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5gw7\" (UniqueName: \"kubernetes.io/projected/0778cc1e-cc9a-473b-a771-7321838515ba-kube-api-access-f5gw7\") pod \"payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx\" (UID: \"0778cc1e-cc9a-473b-a771-7321838515ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"
Oct 30 18:56:06.276462 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.276448    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0778cc1e-cc9a-473b-a771-7321838515ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx\" (UID: \"0778cc1e-cc9a-473b-a771-7321838515ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"
Oct 30 18:56:06.293698 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.293674    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"]
Oct 30 18:56:06.377839 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.377809    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0778cc1e-cc9a-473b-a771-7321838515ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx\" (UID: \"0778cc1e-cc9a-473b-a771-7321838515ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"
Oct 30 18:56:06.377969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.377875    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72dded22-880c-45b7-9a41-381ad30a7f6c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27\" (UID: \"72dded22-880c-45b7-9a41-381ad30a7f6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"
Oct 30 18:56:06.377969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.377909    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5gw7\" (UniqueName: \"kubernetes.io/projected/0778cc1e-cc9a-473b-a771-7321838515ba-kube-api-access-f5gw7\") pod \"payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx\" (UID: \"0778cc1e-cc9a-473b-a771-7321838515ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"
Oct 30 18:56:06.378055 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.377997    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw4g6\" (UniqueName: \"kubernetes.io/projected/72dded22-880c-45b7-9a41-381ad30a7f6c-kube-api-access-gw4g6\") pod \"payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27\" (UID: \"72dded22-880c-45b7-9a41-381ad30a7f6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"
Oct 30 18:56:06.381081 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.381064    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-50b19d\""
Oct 30 18:56:06.388948 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.388922    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0778cc1e-cc9a-473b-a771-7321838515ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx\" (UID: \"0778cc1e-cc9a-473b-a771-7321838515ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"
Oct 30 18:56:06.394273 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.394253    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5gw7\" (UniqueName: \"kubernetes.io/projected/0778cc1e-cc9a-473b-a771-7321838515ba-kube-api-access-f5gw7\") pod \"payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx\" (UID: \"0778cc1e-cc9a-473b-a771-7321838515ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"
Oct 30 18:56:06.492579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.487708    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72dded22-880c-45b7-9a41-381ad30a7f6c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27\" (UID: \"72dded22-880c-45b7-9a41-381ad30a7f6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"
Oct 30 18:56:06.492579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.487846    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw4g6\" (UniqueName: \"kubernetes.io/projected/72dded22-880c-45b7-9a41-381ad30a7f6c-kube-api-access-gw4g6\") pod \"payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27\" (UID: \"72dded22-880c-45b7-9a41-381ad30a7f6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"
Oct 30 18:56:06.492579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.489029    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72dded22-880c-45b7-9a41-381ad30a7f6c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27\" (UID: \"72dded22-880c-45b7-9a41-381ad30a7f6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"
Oct 30 18:56:06.524206 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.517417    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw4g6\" (UniqueName: \"kubernetes.io/projected/72dded22-880c-45b7-9a41-381ad30a7f6c-kube-api-access-gw4g6\") pod \"payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27\" (UID: \"72dded22-880c-45b7-9a41-381ad30a7f6c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"
Oct 30 18:56:06.524448 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.524394    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"
Oct 30 18:56:06.569426 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.564502    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"
Oct 30 18:56:06.984474 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:56:06.984442    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0778cc1e_cc9a_473b_a771_7321838515ba.slice/crio-f7f4f3693307492251b770122ec5ef87d4b032e24182f43ab6f56c1254443729 WatchSource:0}: Error finding container f7f4f3693307492251b770122ec5ef87d4b032e24182f43ab6f56c1254443729: Status 404 returned error can't find the container with id f7f4f3693307492251b770122ec5ef87d4b032e24182f43ab6f56c1254443729
Oct 30 18:56:06.986689 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.986662    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"]
Oct 30 18:56:06.988009 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:56:06.987986    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72dded22_880c_45b7_9a41_381ad30a7f6c.slice/crio-8a137b7f58cc1aeeed2968c52fd13ad2db541382a8dbdb4783b84f6c636cb689 WatchSource:0}: Error finding container 8a137b7f58cc1aeeed2968c52fd13ad2db541382a8dbdb4783b84f6c636cb689: Status 404 returned error can't find the container with id 8a137b7f58cc1aeeed2968c52fd13ad2db541382a8dbdb4783b84f6c636cb689
Oct 30 18:56:06.989969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:06.989950    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"]
Oct 30 18:56:07.227639 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:07.227614    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:56:07.296276 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:07.296200    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27" event={"ID":"72dded22-880c-45b7-9a41-381ad30a7f6c","Type":"ContainerStarted","Data":"5cfc6341c2a704bc9f90a9171af9cfbfa2a5bf3f3c8fb15fce74cab2935ba500"}
Oct 30 18:56:07.296276 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:07.296244    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27" event={"ID":"72dded22-880c-45b7-9a41-381ad30a7f6c","Type":"ContainerStarted","Data":"8a137b7f58cc1aeeed2968c52fd13ad2db541382a8dbdb4783b84f6c636cb689"}
Oct 30 18:56:07.296461 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:07.296413    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"
Oct 30 18:56:07.297952 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:07.297927    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx" event={"ID":"0778cc1e-cc9a-473b-a771-7321838515ba","Type":"ContainerStarted","Data":"9b6ba3ecb86a0933254612b2fdfa971524585514ffc6c4a1082b041151fd1ece"}
Oct 30 18:56:07.298051 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:07.297966    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx" event={"ID":"0778cc1e-cc9a-473b-a771-7321838515ba","Type":"ContainerStarted","Data":"f7f4f3693307492251b770122ec5ef87d4b032e24182f43ab6f56c1254443729"}
Oct 30 18:56:07.298051 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:07.298010    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"
Oct 30 18:56:07.372366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:07.370935    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27" podStartSLOduration=1.370915379 podStartE2EDuration="1.370915379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:07.352540977 +0000 UTC m=+1381.220959308" watchObservedRunningTime="2026-10-30 18:56:07.370915379 +0000 UTC m=+1381.239333715"
Oct 30 18:56:07.396388 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:07.396331    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx" podStartSLOduration=1.396313491 podStartE2EDuration="1.396313491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:07.396196454 +0000 UTC m=+1381.264614785" watchObservedRunningTime="2026-10-30 18:56:07.396313491 +0000 UTC m=+1381.264731821"
Oct 30 18:56:14.927112 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:14.927075    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"]
Oct 30 18:56:14.933839 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:14.933809    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:14.941297 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:14.940146    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-50b19d-serving-cert\""
Oct 30 18:56:14.943303 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:14.943277    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"]
Oct 30 18:56:15.031994 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:15.031958    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dbac43e-9151-4838-928a-f3af21c8f377-metrics-certs\") pod \"maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:15.032163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:15.032109    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dbac43e-9151-4838-928a-f3af21c8f377-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:15.032363 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:15.032341    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dbac43e-9151-4838-928a-f3af21c8f377-maas-api-tls\") pod \"maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:15.032467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:15.032444    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5lqf\" (UniqueName: \"kubernetes.io/projected/1dbac43e-9151-4838-928a-f3af21c8f377-kube-api-access-f5lqf\") pod \"maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:15.133793 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:15.133764    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dbac43e-9151-4838-928a-f3af21c8f377-maas-api-tls\") pod \"maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:15.134039 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:15.134007    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5lqf\" (UniqueName: \"kubernetes.io/projected/1dbac43e-9151-4838-928a-f3af21c8f377-kube-api-access-f5lqf\") pod \"maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:15.134366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:15.134323    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dbac43e-9151-4838-928a-f3af21c8f377-metrics-certs\") pod \"maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:15.141805 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:15.141780    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dbac43e-9151-4838-928a-f3af21c8f377-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:15.173150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:15.159803    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dbac43e-9151-4838-928a-f3af21c8f377-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:15.187317 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:15.187208    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5lqf\" (UniqueName: \"kubernetes.io/projected/1dbac43e-9151-4838-928a-f3af21c8f377-kube-api-access-f5lqf\") pod \"maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:15.210207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:15.209036    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dbac43e-9151-4838-928a-f3af21c8f377-metrics-certs\") pod \"maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:15.248405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:15.234759    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dbac43e-9151-4838-928a-f3af21c8f377-maas-api-tls\") pod \"maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:15.257936 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:15.257904    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:15.549453 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:56:15.549410    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbac43e_9151_4838_928a_f3af21c8f377.slice/crio-a6038aab7cd9c15985b14116876452bba00cbb27901f066983e177e12396df7f WatchSource:0}: Error finding container a6038aab7cd9c15985b14116876452bba00cbb27901f066983e177e12396df7f: Status 404 returned error can't find the container with id a6038aab7cd9c15985b14116876452bba00cbb27901f066983e177e12396df7f
Oct 30 18:56:15.549693 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:15.549670    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"]
Oct 30 18:56:16.476848 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:16.476752    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46" event={"ID":"1dbac43e-9151-4838-928a-f3af21c8f377","Type":"ContainerStarted","Data":"26f69ca6a56b865a09541fa3229cf807efa59748b275994cb2662278853359d6"}
Oct 30 18:56:16.476848 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:16.476830    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46" event={"ID":"1dbac43e-9151-4838-928a-f3af21c8f377","Type":"ContainerStarted","Data":"a6038aab7cd9c15985b14116876452bba00cbb27901f066983e177e12396df7f"}
Oct 30 18:56:16.477326 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:16.476859    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:16.500487 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:16.500428    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46" podStartSLOduration=1.931660988 podStartE2EDuration="2.500414351s" totalImagesPullingTime="568.753363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:16.497182215 +0000 UTC m=+1390.365600542" watchObservedRunningTime="2026-10-30 18:56:16.500414351 +0000 UTC m=+1390.368832682"
Oct 30 18:56:18.392927 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:18.392837    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"
Oct 30 18:56:18.393386 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:18.393104    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"
Oct 30 18:56:20.525468 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:20.525435    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"]
Oct 30 18:56:20.526158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:20.525710    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs" podUID="7a17d135-89a9-4b2e-827a-f5a85b5048a3" containerName="maas-api" containerID="cri-o://16f65aea926e734ca830216f6872a0e7a451b91a7027bfebc8f3b43eb8884fd7" gracePeriod=30
Oct 30 18:56:20.549679 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:20.549639    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs" podUID="7a17d135-89a9-4b2e-827a-f5a85b5048a3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.115:8443/health\": dial tcp 10.133.0.115:8443: connect: connection refused"
Oct 30 18:56:21.248328 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.248300    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:56:21.308305 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.308279    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a17d135-89a9-4b2e-827a-f5a85b5048a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a17d135-89a9-4b2e-827a-f5a85b5048a3-openshift-service-ca\") pod \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") "
Oct 30 18:56:21.308479 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.308359    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a17d135-89a9-4b2e-827a-f5a85b5048a3-kube-api-access-pnswg\" (UniqueName: \"kubernetes.io/projected/7a17d135-89a9-4b2e-827a-f5a85b5048a3-kube-api-access-pnswg\") pod \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") "
Oct 30 18:56:21.308479 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.308386    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a17d135-89a9-4b2e-827a-f5a85b5048a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a17d135-89a9-4b2e-827a-f5a85b5048a3-maas-api-tls\") pod \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") "
Oct 30 18:56:21.308479 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.308480    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a17d135-89a9-4b2e-827a-f5a85b5048a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a17d135-89a9-4b2e-827a-f5a85b5048a3-metrics-certs\") pod \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\" (UID: \"7a17d135-89a9-4b2e-827a-f5a85b5048a3\") "
Oct 30 18:56:21.308744 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.308715    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a17d135-89a9-4b2e-827a-f5a85b5048a3-openshift-service-ca" pod "7a17d135-89a9-4b2e-827a-f5a85b5048a3" (UID: "7a17d135-89a9-4b2e-827a-f5a85b5048a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:21.309014 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.308997    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a17d135-89a9-4b2e-827a-f5a85b5048a3-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:21.310812 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.310781    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a17d135-89a9-4b2e-827a-f5a85b5048a3-metrics-certs" pod "7a17d135-89a9-4b2e-827a-f5a85b5048a3" (UID: "7a17d135-89a9-4b2e-827a-f5a85b5048a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:21.311309 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.311286    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a17d135-89a9-4b2e-827a-f5a85b5048a3-maas-api-tls" pod "7a17d135-89a9-4b2e-827a-f5a85b5048a3" (UID: "7a17d135-89a9-4b2e-827a-f5a85b5048a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:21.311375 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.311346    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a17d135-89a9-4b2e-827a-f5a85b5048a3-kube-api-access-pnswg" pod "7a17d135-89a9-4b2e-827a-f5a85b5048a3" (UID: "7a17d135-89a9-4b2e-827a-f5a85b5048a3"). InnerVolumeSpecName "kube-api-access-pnswg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:21.409988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.409955    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a17d135-89a9-4b2e-827a-f5a85b5048a3-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:21.409988 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.409988    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnswg\" (UniqueName: \"kubernetes.io/projected/7a17d135-89a9-4b2e-827a-f5a85b5048a3-kube-api-access-pnswg\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:21.410221 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.410002    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a17d135-89a9-4b2e-827a-f5a85b5048a3-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:21.510341 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.510264    4372 generic.go:356] "Generic (PLEG): container finished" podID="7a17d135-89a9-4b2e-827a-f5a85b5048a3" containerID="16f65aea926e734ca830216f6872a0e7a451b91a7027bfebc8f3b43eb8884fd7" exitCode=0
Oct 30 18:56:21.510341 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.510323    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"
Oct 30 18:56:21.510504 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.510352    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs" event={"ID":"7a17d135-89a9-4b2e-827a-f5a85b5048a3","Type":"ContainerDied","Data":"16f65aea926e734ca830216f6872a0e7a451b91a7027bfebc8f3b43eb8884fd7"}
Oct 30 18:56:21.510504 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.510420    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs" event={"ID":"7a17d135-89a9-4b2e-827a-f5a85b5048a3","Type":"ContainerDied","Data":"b44b05815f791ee9d7e9a8c5d0ecd509076a24f7095ce0d352ea25815f990d33"}
Oct 30 18:56:21.510504 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.510450    4372 scope.go:122] "RemoveContainer" containerID="16f65aea926e734ca830216f6872a0e7a451b91a7027bfebc8f3b43eb8884fd7"
Oct 30 18:56:21.520997 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.520980    4372 scope.go:122] "RemoveContainer" containerID="16f65aea926e734ca830216f6872a0e7a451b91a7027bfebc8f3b43eb8884fd7"
Oct 30 18:56:21.521298 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:56:21.521278    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f65aea926e734ca830216f6872a0e7a451b91a7027bfebc8f3b43eb8884fd7\": container with ID starting with 16f65aea926e734ca830216f6872a0e7a451b91a7027bfebc8f3b43eb8884fd7 not found: ID does not exist" containerID="16f65aea926e734ca830216f6872a0e7a451b91a7027bfebc8f3b43eb8884fd7"
Oct 30 18:56:21.521342 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.521309    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f65aea926e734ca830216f6872a0e7a451b91a7027bfebc8f3b43eb8884fd7"} err="failed to get container status \"16f65aea926e734ca830216f6872a0e7a451b91a7027bfebc8f3b43eb8884fd7\": rpc error: code = NotFound desc = could not find container \"16f65aea926e734ca830216f6872a0e7a451b91a7027bfebc8f3b43eb8884fd7\": container with ID starting with 16f65aea926e734ca830216f6872a0e7a451b91a7027bfebc8f3b43eb8884fd7 not found: ID does not exist"
Oct 30 18:56:21.544848 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.544822    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"]
Oct 30 18:56:21.547877 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:21.547852    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ceefc-699448c66-s64vs"]
Oct 30 18:56:22.266055 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:22.266023    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a17d135-89a9-4b2e-827a-f5a85b5048a3" path="/var/lib/kubelet/pods/7a17d135-89a9-4b2e-827a-f5a85b5048a3/volumes"
Oct 30 18:56:22.489623 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:22.489601    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:56:26.552360 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:26.552329    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"]
Oct 30 18:56:27.752963 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:56:27.752815    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:56:28.309931 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.309879    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"]
Oct 30 18:56:28.310896 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.310874    4372 cpu_manager.go:414] "containerMap: removing container" podUID="7a17d135-89a9-4b2e-827a-f5a85b5048a3" containerName="maas-api"
Oct 30 18:56:28.311038 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.311026    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a17d135-89a9-4b2e-827a-f5a85b5048a3" containerName="maas-api"
Oct 30 18:56:28.311297 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.311284    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a17d135-89a9-4b2e-827a-f5a85b5048a3" containerName="maas-api"
Oct 30 18:56:28.326318 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.326282    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.340699 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.340676    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"]
Oct 30 18:56:28.345872 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.345843    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.348466 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.348441    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"]
Oct 30 18:56:28.354527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.354502    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"]
Oct 30 18:56:28.436213 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436181    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.436335 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436229    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.436335 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436295    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.436335 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436311    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.436486 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436425    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.436486 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436454    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.436486 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436473    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.436576 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436508    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.436576 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436558    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.436641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436581    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.436641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436619    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.436703 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436648    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.436703 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436685    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.436804 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436787    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8d7p\" (UniqueName: \"kubernetes.io/projected/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-kube-api-access-r8d7p\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.436854 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436841    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.436885 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436872    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsf46\" (UniqueName: \"kubernetes.io/projected/331f4d8f-f859-4c40-a26f-3e6ef07d840a-kube-api-access-lsf46\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.436918 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436911    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.436992 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.436976    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.538203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538172    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.538366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538209    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.538366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538228    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.538366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538248    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.538366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538273    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.538366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538295    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.538366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538316    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.538366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538349    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.538797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538404    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.538797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538453    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8d7p\" (UniqueName: \"kubernetes.io/projected/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-kube-api-access-r8d7p\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.538797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538516    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.538797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538542    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsf46\" (UniqueName: \"kubernetes.io/projected/331f4d8f-f859-4c40-a26f-3e6ef07d840a-kube-api-access-lsf46\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.538797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538584    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.538797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538627    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.538797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538634    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.538797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538685    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.538797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538706    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.538797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538765    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.539467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538812    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.539467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538850    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.539467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.538958    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.539467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.539158    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.539467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.539396    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.539467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.539442    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.539850 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.539825    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.539958 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.539848    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.539958 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.539945    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.540110 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.540068    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.541225 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.541199    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.541344 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.541326    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.541506 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.541487    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.541992 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.541975    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.555647 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.555618    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsf46\" (UniqueName: \"kubernetes.io/projected/331f4d8f-f859-4c40-a26f-3e6ef07d840a-kube-api-access-lsf46\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.556713 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.556691    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.561251 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.561203    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.611845 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.611810    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8d7p\" (UniqueName: \"kubernetes.io/projected/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-kube-api-access-r8d7p\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.671245 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.671216    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:28.674331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.674310    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:28.962623 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.962595    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"]
Oct 30 18:56:28.964092 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:56:28.964057    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b49aa4_fbad_4b2d_91b0_bece668fbfd3.slice/crio-6451c99617ee8a1da18ca730ac6c52597ed6f5fdb667b4b127d4cb992426396b WatchSource:0}: Error finding container 6451c99617ee8a1da18ca730ac6c52597ed6f5fdb667b4b127d4cb992426396b: Status 404 returned error can't find the container with id 6451c99617ee8a1da18ca730ac6c52597ed6f5fdb667b4b127d4cb992426396b
Oct 30 18:56:28.965274 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:56:28.965252    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331f4d8f_f859_4c40_a26f_3e6ef07d840a.slice/crio-3a72f3480fcd74a315a1c5a2ed775a9e488e900845c31ca79f8605bdc2915dff WatchSource:0}: Error finding container 3a72f3480fcd74a315a1c5a2ed775a9e488e900845c31ca79f8605bdc2915dff: Status 404 returned error can't find the container with id 3a72f3480fcd74a315a1c5a2ed775a9e488e900845c31ca79f8605bdc2915dff
Oct 30 18:56:28.966410 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.966389    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"]
Oct 30 18:56:28.967294 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.967258    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:28.967382 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.967365    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:28.967431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.967378    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:28.967431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.967411    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:28.967498 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.967442    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:28.967498 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:28.967472    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:29.563605 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:29.563563    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" event={"ID":"76b49aa4-fbad-4b2d-91b0-bece668fbfd3","Type":"ContainerStarted","Data":"6f0a9e869f2f9e60e06c8c2e6e5ee23a60ae7299fafbc645bb68f042cbc3d4fd"}
Oct 30 18:56:29.563785 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:29.563624    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" event={"ID":"76b49aa4-fbad-4b2d-91b0-bece668fbfd3","Type":"ContainerStarted","Data":"6451c99617ee8a1da18ca730ac6c52597ed6f5fdb667b4b127d4cb992426396b"}
Oct 30 18:56:29.565278 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:29.565249    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" event={"ID":"331f4d8f-f859-4c40-a26f-3e6ef07d840a","Type":"ContainerStarted","Data":"6627b03ea5bee45d11b62146f879ba00305e251de7a63603474c31bec7b886cf"}
Oct 30 18:56:29.565388 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:29.565299    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" event={"ID":"331f4d8f-f859-4c40-a26f-3e6ef07d840a","Type":"ContainerStarted","Data":"3a72f3480fcd74a315a1c5a2ed775a9e488e900845c31ca79f8605bdc2915dff"}
Oct 30 18:56:29.591670 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:29.591622    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" podStartSLOduration=1.59160741 podStartE2EDuration="1.59160741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:29.588577715 +0000 UTC m=+1403.456996045" watchObservedRunningTime="2026-10-30 18:56:29.59160741 +0000 UTC m=+1403.460025740"
Oct 30 18:56:29.617370 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:29.617232    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" podStartSLOduration=1.617213856 podStartE2EDuration="1.617213856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:29.611196673 +0000 UTC m=+1403.479615006" watchObservedRunningTime="2026-10-30 18:56:29.617213856 +0000 UTC m=+1403.485632187"
Oct 30 18:56:29.672096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:29.672073    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:29.675204 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:29.675185    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:29.942485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:29.928190    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"]
Oct 30 18:56:29.942485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:29.942116    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"]
Oct 30 18:56:30.672946 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:30.672900    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 18:56:30.673429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:30.672965    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" podUID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 18:56:30.676146 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:30.676106    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:56:30.676207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:30.676177    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" podUID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 18:56:30.947204 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:30.946028    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"]
Oct 30 18:56:30.951274 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:30.951248    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:30.953673 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:30.953649    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"]
Oct 30 18:56:31.050197 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.050160    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.050428 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.050232    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.050428 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.050260    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdzjn\" (UniqueName: \"kubernetes.io/projected/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-kube-api-access-rdzjn\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.050428 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.050292    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.050428 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.050378    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.050428 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.050407    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.051883 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.050457    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.051883 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.050497    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.051883 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.050598    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.152081 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.152045    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.152275 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.152169    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.152275 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.152217    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.152275 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.152234    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdzjn\" (UniqueName: \"kubernetes.io/projected/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-kube-api-access-rdzjn\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.152275 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.152259    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.152414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.152317    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.152414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.152339    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.152414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.152374    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.152414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.152405    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.152569 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.152548    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.152755 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.152738    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.152897 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.152876    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.153425 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.153392    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.153570 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.153463    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.154730 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.154710    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.155086 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.155065    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.166178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.166154    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdzjn\" (UniqueName: \"kubernetes.io/projected/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-kube-api-access-rdzjn\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.166859 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.166839    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.274201 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.273380    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:31.750154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.718273    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:56:31.750154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.718350    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" podUID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 18:56:31.750154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.718790    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 18:56:31.750154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:31.718842    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" podUID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 18:56:32.097151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:32.095901    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"]
Oct 30 18:56:32.140797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:32.140755    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:32.141015 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:32.140993    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:32.141180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:32.141158    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:32.686502 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:32.686414    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:56:32.686843 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:32.686801    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" podUID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 18:56:32.706984 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:32.706954    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 18:56:32.709866 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:32.709826    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" podUID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 18:56:32.717337 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:32.708583    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" event={"ID":"7ce78af1-d7d9-4019-ab03-aa84fda13e1c","Type":"ContainerStarted","Data":"58a14207ece4febcdef46fcbb504f6ced2a1370b81c4b7582bba42a8fc8197bb"}
Oct 30 18:56:32.728440 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:32.728409    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" event={"ID":"7ce78af1-d7d9-4019-ab03-aa84fda13e1c","Type":"ContainerStarted","Data":"ec57f782c5db70b308a92709aebb7ca9fb69d955dee576ff7de98ffc444ecd5d"}
Oct 30 18:56:32.828516 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:32.828450    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" podStartSLOduration=2.828426854 podStartE2EDuration="2.828426854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:32.751575677 +0000 UTC m=+1406.619994009" watchObservedRunningTime="2026-10-30 18:56:32.828426854 +0000 UTC m=+1406.696845184"
Oct 30 18:56:33.298152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:33.295100    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:33.298152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:33.295357    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:56:33.298152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:33.295411    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:56:33.675152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:33.674597    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 18:56:33.675152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:33.674661    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" podUID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 18:56:33.675435 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:33.675400    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:56:33.675511 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:33.675445    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" podUID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 18:56:33.921153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:33.916320    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:33.962334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:33.958231    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 18:56:33.982154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:33.974345    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:34.070151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:34.062211    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:34.277211 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:34.276559    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:56:34.277211 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:34.276626    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:56:34.444152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:34.443178    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:34.484155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:34.483265    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": context deadline exceeded"
Oct 30 18:56:34.681172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:34.674656    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 18:56:34.681172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:34.674721    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" podUID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 18:56:34.697154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:34.681567    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:56:34.697154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:34.681621    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" podUID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 18:56:34.935208 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:34.928488    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:34.935208 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:34.929386    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:56:34.980309 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:34.942247    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27" podUID="72dded22-880c-45b7-9a41-381ad30a7f6c" containerName="payload-processing" containerID="cri-o://5cfc6341c2a704bc9f90a9171af9cfbfa2a5bf3f3c8fb15fce74cab2935ba500" gracePeriod=25
Oct 30 18:56:34.980309 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:34.942431    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx" podUID="0778cc1e-cc9a-473b-a771-7321838515ba" containerName="payload-pre-processing" containerID="cri-o://9b6ba3ecb86a0933254612b2fdfa971524585514ffc6c4a1082b041151fd1ece" gracePeriod=25
Oct 30 18:56:34.980309 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:34.958815    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:34.980309 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:34.959521    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:56:35.056747 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.056156    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 18:56:35.078733 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.058225    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:56:35.080534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.079731    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:35.100153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.081708    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:56:35.537176 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.516237    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:56:35.537176 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.516311    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:56:35.537176 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.517737    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 18:56:35.560825 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.560788    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:56:35.564250 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.564221    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:56:35.590107 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.574954    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": context deadline exceeded"
Oct 30 18:56:35.590107 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.575525    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:56:35.635334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.634323    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc" podUID="a68833d6-91b3-4c78-9115-aac5f70dbba8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.129:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:35.690432 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.690224    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:56:35.690432 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.690340    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" podUID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 18:56:35.693031 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.690742    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 18:56:35.693031 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.690781    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" podUID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 18:56:35.909151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.854775    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:35.954325 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.954282    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 18:56:35.982660 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.971878    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:36.096747 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:35.971922    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:36.096747 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:36.082946    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:36.392174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:36.382292    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:56:36.392174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:36.382356    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:56:36.538593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:36.538491    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:36.636189 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:36.591627    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:36.636945 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:36.636901    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc" podUID="a68833d6-91b3-4c78-9115-aac5f70dbba8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.129:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:36.638678 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:36.638653    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:56:36.850253 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:36.848804    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 18:56:36.850253 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:36.848877    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" podUID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 18:56:36.850253 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:36.849390    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:56:36.850253 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:36.849434    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" podUID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 18:56:37.050476 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.050404    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:37.051826 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.051794    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:37.052083 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.052061    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:37.292147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.288455    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 18:56:37.438660 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.350250    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:56:37.483938 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.468152    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:56:37.503524 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.468671    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:37.597194 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.595680    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:37.694169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.628564    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:56:37.694169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.628627    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:56:37.694169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.632006    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:37.694169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.666083    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc" podUID="a68833d6-91b3-4c78-9115-aac5f70dbba8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.129:8000/ready\": context deadline exceeded"
Oct 30 18:56:37.711012 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.710968    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 18:56:37.757764 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.711285    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" podUID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 18:56:37.785678 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.757449    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:56:37.785678 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.776186    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" podUID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 18:56:37.785678 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.777725    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-9mxbq" podUID="0316c797-de93-4782-9506-7057fb3aa4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:37.960703 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.960612    4372 generic.go:356] "Generic (PLEG): container finished" podID="72dded22-880c-45b7-9a41-381ad30a7f6c" containerID="5cfc6341c2a704bc9f90a9171af9cfbfa2a5bf3f3c8fb15fce74cab2935ba500" exitCode=0
Oct 30 18:56:37.961032 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:37.961001    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27" event={"ID":"72dded22-880c-45b7-9a41-381ad30a7f6c","Type":"ContainerDied","Data":"5cfc6341c2a704bc9f90a9171af9cfbfa2a5bf3f3c8fb15fce74cab2935ba500"}
Oct 30 18:56:38.158151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.153460    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"]
Oct 30 18:56:38.159413 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.158960    4372 generic.go:356] "Generic (PLEG): container finished" podID="0778cc1e-cc9a-473b-a771-7321838515ba" containerID="9b6ba3ecb86a0933254612b2fdfa971524585514ffc6c4a1082b041151fd1ece" exitCode=0
Oct 30 18:56:38.263649 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.262570    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 18:56:38.279241 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.266336    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"
Oct 30 18:56:38.283218 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.280223    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:56:38.283218 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.280287    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:56:38.287346 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.287321    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"
Oct 30 18:56:38.372283 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.306398    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 18:56:38.372283 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.307497    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"
Oct 30 18:56:38.372283 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.350324    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:38.372283 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.350694    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:38.372283 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.351260    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded"
Oct 30 18:56:38.480167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.466549    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72dded22-880c-45b7-9a41-381ad30a7f6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72dded22-880c-45b7-9a41-381ad30a7f6c-plugins-config-volume\") pod \"72dded22-880c-45b7-9a41-381ad30a7f6c\" (UID: \"72dded22-880c-45b7-9a41-381ad30a7f6c\") "
Oct 30 18:56:38.480167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.466604    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72dded22-880c-45b7-9a41-381ad30a7f6c-kube-api-access-gw4g6\" (UniqueName: \"kubernetes.io/projected/72dded22-880c-45b7-9a41-381ad30a7f6c-kube-api-access-gw4g6\") pod \"72dded22-880c-45b7-9a41-381ad30a7f6c\" (UID: \"72dded22-880c-45b7-9a41-381ad30a7f6c\") "
Oct 30 18:56:38.480167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.466677    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0778cc1e-cc9a-473b-a771-7321838515ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0778cc1e-cc9a-473b-a771-7321838515ba-plugins-config-volume\") pod \"0778cc1e-cc9a-473b-a771-7321838515ba\" (UID: \"0778cc1e-cc9a-473b-a771-7321838515ba\") "
Oct 30 18:56:38.480167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.466946    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0778cc1e-cc9a-473b-a771-7321838515ba-kube-api-access-f5gw7\" (UniqueName: \"kubernetes.io/projected/0778cc1e-cc9a-473b-a771-7321838515ba-kube-api-access-f5gw7\") pod \"0778cc1e-cc9a-473b-a771-7321838515ba\" (UID: \"0778cc1e-cc9a-473b-a771-7321838515ba\") "
Oct 30 18:56:38.480167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.467363    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xhpg\" (UniqueName: \"kubernetes.io/projected/1da28a54-d60a-49c7-b8ff-f2ca79f5636f-kube-api-access-9xhpg\") pod \"payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg\" (UID: \"1da28a54-d60a-49c7-b8ff-f2ca79f5636f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"
Oct 30 18:56:38.480167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.467430    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da28a54-d60a-49c7-b8ff-f2ca79f5636f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg\" (UID: \"1da28a54-d60a-49c7-b8ff-f2ca79f5636f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"
Oct 30 18:56:38.480167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.472449    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72dded22-880c-45b7-9a41-381ad30a7f6c-plugins-config-volume" pod "72dded22-880c-45b7-9a41-381ad30a7f6c" (UID: "72dded22-880c-45b7-9a41-381ad30a7f6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:38.480167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.472888    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0778cc1e-cc9a-473b-a771-7321838515ba-plugins-config-volume" pod "0778cc1e-cc9a-473b-a771-7321838515ba" (UID: "0778cc1e-cc9a-473b-a771-7321838515ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:38.559148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.557858    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0778cc1e-cc9a-473b-a771-7321838515ba-kube-api-access-f5gw7" pod "0778cc1e-cc9a-473b-a771-7321838515ba" (UID: "0778cc1e-cc9a-473b-a771-7321838515ba"). InnerVolumeSpecName "kube-api-access-f5gw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:38.570154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.569670    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xhpg\" (UniqueName: \"kubernetes.io/projected/1da28a54-d60a-49c7-b8ff-f2ca79f5636f-kube-api-access-9xhpg\") pod \"payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg\" (UID: \"1da28a54-d60a-49c7-b8ff-f2ca79f5636f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"
Oct 30 18:56:38.570451 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.570426    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da28a54-d60a-49c7-b8ff-f2ca79f5636f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg\" (UID: \"1da28a54-d60a-49c7-b8ff-f2ca79f5636f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"
Oct 30 18:56:38.570863 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.570847    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5gw7\" (UniqueName: \"kubernetes.io/projected/0778cc1e-cc9a-473b-a771-7321838515ba-kube-api-access-f5gw7\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:38.570957 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.570946    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72dded22-880c-45b7-9a41-381ad30a7f6c-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:38.571042 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.571032    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0778cc1e-cc9a-473b-a771-7321838515ba-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:38.571934 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.571913    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da28a54-d60a-49c7-b8ff-f2ca79f5636f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg\" (UID: \"1da28a54-d60a-49c7-b8ff-f2ca79f5636f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"
Oct 30 18:56:38.574646 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.574622    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:56:38.574769 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.574733    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx" event={"ID":"0778cc1e-cc9a-473b-a771-7321838515ba","Type":"ContainerDied","Data":"9b6ba3ecb86a0933254612b2fdfa971524585514ffc6c4a1082b041151fd1ece"}
Oct 30 18:56:38.574769 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.574764    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx" event={"ID":"0778cc1e-cc9a-473b-a771-7321838515ba","Type":"ContainerDied","Data":"f7f4f3693307492251b770122ec5ef87d4b032e24182f43ab6f56c1254443729"}
Oct 30 18:56:38.574892 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.574791    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"]
Oct 30 18:56:38.574892 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.574808    4372 scope.go:122] "RemoveContainer" containerID="9b6ba3ecb86a0933254612b2fdfa971524585514ffc6c4a1082b041151fd1ece"
Oct 30 18:56:38.575487 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.575469    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"]
Oct 30 18:56:38.576472 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.576452    4372 cpu_manager.go:414] "containerMap: removing container" podUID="0778cc1e-cc9a-473b-a771-7321838515ba" containerName="payload-pre-processing"
Oct 30 18:56:38.601341 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.601317    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0778cc1e-cc9a-473b-a771-7321838515ba" containerName="payload-pre-processing"
Oct 30 18:56:38.601524 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.601511    4372 cpu_manager.go:414] "containerMap: removing container" podUID="72dded22-880c-45b7-9a41-381ad30a7f6c" containerName="payload-processing"
Oct 30 18:56:38.601613 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.601601    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72dded22-880c-45b7-9a41-381ad30a7f6c" containerName="payload-processing"
Oct 30 18:56:38.601957 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.601937    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="0778cc1e-cc9a-473b-a771-7321838515ba" containerName="payload-pre-processing"
Oct 30 18:56:38.602306 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.602289    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="72dded22-880c-45b7-9a41-381ad30a7f6c" containerName="payload-processing"
Oct 30 18:56:38.615022 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.614973    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72dded22-880c-45b7-9a41-381ad30a7f6c-kube-api-access-gw4g6" pod "72dded22-880c-45b7-9a41-381ad30a7f6c" (UID: "72dded22-880c-45b7-9a41-381ad30a7f6c"). InnerVolumeSpecName "kube-api-access-gw4g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:38.666056 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.648682    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:38.666056 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.658202    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 18:56:38.675439 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.675406    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 18:56:38.724315 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.724271    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw4g6\" (UniqueName: \"kubernetes.io/projected/72dded22-880c-45b7-9a41-381ad30a7f6c-kube-api-access-gw4g6\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:38.733173 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.728200    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:38.740901 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.740872    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:56:38.741096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.741072    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" podUID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 18:56:38.763104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.763023    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 18:56:38.763104 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.763079    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" podUID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 18:56:38.764760 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.764738    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Oct 30 18:56:38.871170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.869506    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:56:38.871170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.869565    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"]
Oct 30 18:56:38.871170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.870884    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"
Oct 30 18:56:38.907609 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.907556    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-50b19d-dockercfg-gw4qk\""
Oct 30 18:56:38.928329 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.928281    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9pzl\" (UniqueName: \"kubernetes.io/projected/daef0fb1-64c9-4e07-aaa3-66eb205cc648-kube-api-access-s9pzl\") pod \"payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd\" (UID: \"daef0fb1-64c9-4e07-aaa3-66eb205cc648\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"
Oct 30 18:56:38.928795 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:38.928773    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daef0fb1-64c9-4e07-aaa3-66eb205cc648-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd\" (UID: \"daef0fb1-64c9-4e07-aaa3-66eb205cc648\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"
Oct 30 18:56:39.043997 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.042188    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daef0fb1-64c9-4e07-aaa3-66eb205cc648-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd\" (UID: \"daef0fb1-64c9-4e07-aaa3-66eb205cc648\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"
Oct 30 18:56:39.043997 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.042435    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9pzl\" (UniqueName: \"kubernetes.io/projected/daef0fb1-64c9-4e07-aaa3-66eb205cc648-kube-api-access-s9pzl\") pod \"payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd\" (UID: \"daef0fb1-64c9-4e07-aaa3-66eb205cc648\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"
Oct 30 18:56:39.043997 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.043893    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daef0fb1-64c9-4e07-aaa3-66eb205cc648-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd\" (UID: \"daef0fb1-64c9-4e07-aaa3-66eb205cc648\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"
Oct 30 18:56:39.071623 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.065306    4372 scope.go:122] "RemoveContainer" containerID="9b6ba3ecb86a0933254612b2fdfa971524585514ffc6c4a1082b041151fd1ece"
Oct 30 18:56:39.072866 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:56:39.072033    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b6ba3ecb86a0933254612b2fdfa971524585514ffc6c4a1082b041151fd1ece\": container with ID starting with 9b6ba3ecb86a0933254612b2fdfa971524585514ffc6c4a1082b041151fd1ece not found: ID does not exist" containerID="9b6ba3ecb86a0933254612b2fdfa971524585514ffc6c4a1082b041151fd1ece"
Oct 30 18:56:39.072866 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.072078    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b6ba3ecb86a0933254612b2fdfa971524585514ffc6c4a1082b041151fd1ece"} err="failed to get container status \"9b6ba3ecb86a0933254612b2fdfa971524585514ffc6c4a1082b041151fd1ece\": rpc error: code = NotFound desc = could not find container \"9b6ba3ecb86a0933254612b2fdfa971524585514ffc6c4a1082b041151fd1ece\": container with ID starting with 9b6ba3ecb86a0933254612b2fdfa971524585514ffc6c4a1082b041151fd1ece not found: ID does not exist"
Oct 30 18:56:39.116735 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.088317    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xhpg\" (UniqueName: \"kubernetes.io/projected/1da28a54-d60a-49c7-b8ff-f2ca79f5636f-kube-api-access-9xhpg\") pod \"payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg\" (UID: \"1da28a54-d60a-49c7-b8ff-f2ca79f5636f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"
Oct 30 18:56:39.127179 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.117267    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:56:39.127179 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.122484    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"
Oct 30 18:56:39.167681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.167655    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:56:39.176826 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.176791    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9pzl\" (UniqueName: \"kubernetes.io/projected/daef0fb1-64c9-4e07-aaa3-66eb205cc648-kube-api-access-s9pzl\") pod \"payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd\" (UID: \"daef0fb1-64c9-4e07-aaa3-66eb205cc648\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"
Oct 30 18:56:39.368500 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.358341    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"
Oct 30 18:56:39.375221 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.375194    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:56:39.381909 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.378435    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:56:39.381909 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.378492    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:56:39.461151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.451544    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:39.501149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.499558    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27" event={"ID":"72dded22-880c-45b7-9a41-381ad30a7f6c","Type":"ContainerDied","Data":"8a137b7f58cc1aeeed2968c52fd13ad2db541382a8dbdb4783b84f6c636cb689"}
Oct 30 18:56:39.501149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.499633    4372 scope.go:122] "RemoveContainer" containerID="5cfc6341c2a704bc9f90a9171af9cfbfa2a5bf3f3c8fb15fce74cab2935ba500"
Oct 30 18:56:39.508151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.502334    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:56:39.528162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.523207    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"
Oct 30 18:56:39.589321 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.589048    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:56:39.672316 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.668447    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:56:39.676331 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.676304    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 18:56:39.696712 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.696669    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" podUID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 18:56:39.739663 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.739615    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"
Oct 30 18:56:39.740861 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.740829    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 18:56:39.740985 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:39.740883    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" podUID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 18:56:40.148287 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:40.148198    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"]
Oct 30 18:56:40.284732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:40.284695    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:56:40.285336 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:40.285303    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:56:41.000177 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:40.987060    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:41.277752 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:41.277713    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b559fc78-nrb27"]
Oct 30 18:56:41.277897 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:41.277862    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:41.277989 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:41.277919    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"]
Oct 30 18:56:41.277989 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:41.277945    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-59c98d8547-zbdfx"]
Oct 30 18:56:41.392273 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:41.378380    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:56:41.392273 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:41.378443    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:56:41.417172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:41.409632    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:41.427151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:41.417420    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 18:56:41.513155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:41.466658    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:41.513155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:41.469515    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:41.513155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:41.471176    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 18:56:41.574500 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:41.574208    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc" podUID="a68833d6-91b3-4c78-9115-aac5f70dbba8" containerName="main" containerID="cri-o://e6ea3dc8d6a1eace39fb14dae0cc25a09621bf537b5c920ea16ccbb64f8b0deb" gracePeriod=45
Oct 30 18:56:41.659838 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:41.659698    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc" podUID="a68833d6-91b3-4c78-9115-aac5f70dbba8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.129:8000/ready\": dial tcp 10.133.0.129:8000: connect: connection refused"
Oct 30 18:56:41.670302 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:41.670261    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"]
Oct 30 18:56:42.070013 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.056585    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"]
Oct 30 18:56:42.274936 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.274085    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:56:42.274936 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.274167    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:56:42.298761 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.295899    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0778cc1e-cc9a-473b-a771-7321838515ba" path="/var/lib/kubelet/pods/0778cc1e-cc9a-473b-a771-7321838515ba/volumes"
Oct 30 18:56:42.298761 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.296854    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72dded22-880c-45b7-9a41-381ad30a7f6c" path="/var/lib/kubelet/pods/72dded22-880c-45b7-9a41-381ad30a7f6c/volumes"
Oct 30 18:56:42.394914 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.394572    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:56:42.699412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.698040    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg" event={"ID":"1da28a54-d60a-49c7-b8ff-f2ca79f5636f","Type":"ContainerStarted","Data":"f47e3d49c81dfe9d34c14b4fa51e62650bc0aac481137da501f0323cdb6617ff"}
Oct 30 18:56:42.699412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.698109    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg" event={"ID":"1da28a54-d60a-49c7-b8ff-f2ca79f5636f","Type":"ContainerStarted","Data":"2c028c93ea046d6b73f95a2cd497c93e8fbe98d655d7bb047d520229707201dd"}
Oct 30 18:56:42.699412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.699350    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"
Oct 30 18:56:42.723113 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.723078    4372 generic.go:356] "Generic (PLEG): container finished" podID="a68833d6-91b3-4c78-9115-aac5f70dbba8" containerID="e6ea3dc8d6a1eace39fb14dae0cc25a09621bf537b5c920ea16ccbb64f8b0deb" exitCode=0
Oct 30 18:56:42.746546 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.724263    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"
Oct 30 18:56:42.779437 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.724286    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc" event={"ID":"a68833d6-91b3-4c78-9115-aac5f70dbba8","Type":"ContainerDied","Data":"e6ea3dc8d6a1eace39fb14dae0cc25a09621bf537b5c920ea16ccbb64f8b0deb"}
Oct 30 18:56:42.779710 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.779679    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc" event={"ID":"a68833d6-91b3-4c78-9115-aac5f70dbba8","Type":"ContainerDied","Data":"2daf6e2f80537d6be52ff645eb69841853e0bba6eef661eb106440196ba3db8d"}
Oct 30 18:56:42.779865 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.779811    4372 scope.go:122] "RemoveContainer" containerID="e6ea3dc8d6a1eace39fb14dae0cc25a09621bf537b5c920ea16ccbb64f8b0deb"
Oct 30 18:56:42.828176 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.803959    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-home\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-home\") pod \"a68833d6-91b3-4c78-9115-aac5f70dbba8\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") "
Oct 30 18:56:42.828176 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.804058    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-dshm\") pod \"a68833d6-91b3-4c78-9115-aac5f70dbba8\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") "
Oct 30 18:56:42.828176 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.804380    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-home" pod "a68833d6-91b3-4c78-9115-aac5f70dbba8" (UID: "a68833d6-91b3-4c78-9115-aac5f70dbba8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:42.879560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.842229    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a68833d6-91b3-4c78-9115-aac5f70dbba8-kube-api-access-s7wk5\" (UniqueName: \"kubernetes.io/projected/a68833d6-91b3-4c78-9115-aac5f70dbba8-kube-api-access-s7wk5\") pod \"a68833d6-91b3-4c78-9115-aac5f70dbba8\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") "
Oct 30 18:56:42.879560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.842350    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-tmp-dir\") pod \"a68833d6-91b3-4c78-9115-aac5f70dbba8\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") "
Oct 30 18:56:42.879560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.842465    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-model-cache\") pod \"a68833d6-91b3-4c78-9115-aac5f70dbba8\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") "
Oct 30 18:56:42.879560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.842532    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a68833d6-91b3-4c78-9115-aac5f70dbba8-tls-certs\" (UniqueName: \"kubernetes.io/secret/a68833d6-91b3-4c78-9115-aac5f70dbba8-tls-certs\") pod \"a68833d6-91b3-4c78-9115-aac5f70dbba8\" (UID: \"a68833d6-91b3-4c78-9115-aac5f70dbba8\") "
Oct 30 18:56:42.879560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.843314    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:42.879560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.858209    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-model-cache" pod "a68833d6-91b3-4c78-9115-aac5f70dbba8" (UID: "a68833d6-91b3-4c78-9115-aac5f70dbba8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:42.879560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.858418    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-tmp-dir" pod "a68833d6-91b3-4c78-9115-aac5f70dbba8" (UID: "a68833d6-91b3-4c78-9115-aac5f70dbba8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:42.908398 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.904674    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68833d6-91b3-4c78-9115-aac5f70dbba8-tls-certs" pod "a68833d6-91b3-4c78-9115-aac5f70dbba8" (UID: "a68833d6-91b3-4c78-9115-aac5f70dbba8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:42.908398 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.904759    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-dshm" pod "a68833d6-91b3-4c78-9115-aac5f70dbba8" (UID: "a68833d6-91b3-4c78-9115-aac5f70dbba8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:42.913724 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.913651    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68833d6-91b3-4c78-9115-aac5f70dbba8-kube-api-access-s7wk5" pod "a68833d6-91b3-4c78-9115-aac5f70dbba8" (UID: "a68833d6-91b3-4c78-9115-aac5f70dbba8"). InnerVolumeSpecName "kube-api-access-s7wk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:42.944641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.944491    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7wk5\" (UniqueName: \"kubernetes.io/projected/a68833d6-91b3-4c78-9115-aac5f70dbba8-kube-api-access-s7wk5\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:42.944641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.944526    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:42.944641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.944545    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:42.944641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.944560    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68833d6-91b3-4c78-9115-aac5f70dbba8-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:42.944641 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.944575    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68833d6-91b3-4c78-9115-aac5f70dbba8-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:42.980148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.976967    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd" event={"ID":"daef0fb1-64c9-4e07-aaa3-66eb205cc648","Type":"ContainerStarted","Data":"8c5743fd80fdc5ad8f2ca00eb4b3cfe8466a19d7b47c5a3d13ebf869045c460b"}
Oct 30 18:56:42.980148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.977021    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd" event={"ID":"daef0fb1-64c9-4e07-aaa3-66eb205cc648","Type":"ContainerStarted","Data":"f9fe1c5177dd60e478ed1a3d9c57461b2da89f2ef599de83829fe432d3411143"}
Oct 30 18:56:42.980148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:42.977598    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"
Oct 30 18:56:43.005297 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:43.002299    4372 scope.go:122] "RemoveContainer" containerID="e6ea3dc8d6a1eace39fb14dae0cc25a09621bf537b5c920ea16ccbb64f8b0deb"
Oct 30 18:56:43.068147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:43.008674    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg" podStartSLOduration=6.008655722 podStartE2EDuration="6.008655722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:42.858669099 +0000 UTC m=+1416.727087434" watchObservedRunningTime="2026-10-30 18:56:43.008655722 +0000 UTC m=+1416.877074054"
Oct 30 18:56:43.068307 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:56:43.067524    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ea3dc8d6a1eace39fb14dae0cc25a09621bf537b5c920ea16ccbb64f8b0deb\": container with ID starting with e6ea3dc8d6a1eace39fb14dae0cc25a09621bf537b5c920ea16ccbb64f8b0deb not found: ID does not exist" containerID="e6ea3dc8d6a1eace39fb14dae0cc25a09621bf537b5c920ea16ccbb64f8b0deb"
Oct 30 18:56:43.068307 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:43.068257    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ea3dc8d6a1eace39fb14dae0cc25a09621bf537b5c920ea16ccbb64f8b0deb"} err="failed to get container status \"e6ea3dc8d6a1eace39fb14dae0cc25a09621bf537b5c920ea16ccbb64f8b0deb\": rpc error: code = NotFound desc = could not find container \"e6ea3dc8d6a1eace39fb14dae0cc25a09621bf537b5c920ea16ccbb64f8b0deb\": container with ID starting with e6ea3dc8d6a1eace39fb14dae0cc25a09621bf537b5c920ea16ccbb64f8b0deb not found: ID does not exist"
Oct 30 18:56:43.068893 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:43.068839    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd" podStartSLOduration=6.068825782 podStartE2EDuration="6.068825782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:43.005698832 +0000 UTC m=+1416.874117173" watchObservedRunningTime="2026-10-30 18:56:43.068825782 +0000 UTC m=+1416.937244112"
Oct 30 18:56:43.224443 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:43.224408    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"]
Oct 30 18:56:43.235977 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:43.235942    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-9ceefc/ipp-route-9ceefc-kserve-6c55f4b797-gzzdc"]
Oct 30 18:56:43.274207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:43.274178    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:56:43.274322 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:43.274231    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:56:44.268943 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:44.268894    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a68833d6-91b3-4c78-9115-aac5f70dbba8" path="/var/lib/kubelet/pods/a68833d6-91b3-4c78-9115-aac5f70dbba8/volumes"
Oct 30 18:56:44.279813 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:44.279784    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:44.280276 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:44.280243    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:44.281066 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:44.281035    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 18:56:48.034000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:48.033969    4372 generic.go:356] "Generic (PLEG): container finished" podID="c448eb1a-dfec-4669-a5e7-585087730fbc" containerID="9c76113f3c511d2323e748ba0a7d742c6bfbc6d9e77738b9d827f7c4e27fc626" exitCode=0
Oct 30 18:56:48.034447 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:48.034033    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5" event={"ID":"c448eb1a-dfec-4669-a5e7-585087730fbc","Type":"ContainerDied","Data":"9c76113f3c511d2323e748ba0a7d742c6bfbc6d9e77738b9d827f7c4e27fc626"}
Oct 30 18:56:48.034499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:48.034483    4372 scope.go:122] "RemoveContainer" containerID="9c76113f3c511d2323e748ba0a7d742c6bfbc6d9e77738b9d827f7c4e27fc626"
Oct 30 18:56:48.035969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:48.035941    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:48.036038 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:48.036022    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:48.036165 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:48.036056    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:56:48.269238 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:48.269207    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:49.042235 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:49.042193    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5" event={"ID":"c448eb1a-dfec-4669-a5e7-585087730fbc","Type":"ContainerStarted","Data":"ef37a6325d5e2820bab4b28935f17235178346e364ef6ad808a70e2296dcb057"}
Oct 30 18:56:49.363523 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:49.363493    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:49.364517 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:49.364493    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 18:56:49.364643 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:49.364541    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5" podUID="c448eb1a-dfec-4669-a5e7-585087730fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 18:56:50.364023 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:50.363995    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 18:56:50.364451 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:50.364058    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5" podUID="c448eb1a-dfec-4669-a5e7-585087730fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 18:56:51.364412 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:51.364378    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 18:56:51.364814 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:51.364437    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5" podUID="c448eb1a-dfec-4669-a5e7-585087730fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 18:56:52.369370 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:52.369334    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 18:56:52.369994 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:52.369962    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5" podUID="c448eb1a-dfec-4669-a5e7-585087730fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 18:56:53.363571 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:53.363537    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 18:56:53.363760 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:53.363597    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5" podUID="c448eb1a-dfec-4669-a5e7-585087730fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 18:56:53.558319 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:53.558284    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"]
Oct 30 18:56:53.643377 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:53.643177    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"]
Oct 30 18:56:54.370277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:54.370247    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:54.370582 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:54.370557    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:54.371674 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:54.371652    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:56:54.998155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:54.998095    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"
Oct 30 18:56:54.998647 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:54.998334    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"
Oct 30 18:56:56.926017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:56.925983    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:56:57.880228 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:56:57.868321    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:56:58.563420 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:58.563379    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9" podUID="e2f7d978-361f-4300-869e-985a9ff73cec" containerName="payload-processing" containerID="cri-o://dc1c6ce3b8f4bfe5768733417af250c61dcb251fdbfafd860d2619b58dca5c31" gracePeriod=25
Oct 30 18:56:58.647369 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:58.647187    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54" podUID="136d8931-c397-4378-a8eb-c33842cf0776" containerName="payload-pre-processing" containerID="cri-o://3de0c7b54a6948eb777485a6b5f946f9ed869763861349908edc8c023400138b" gracePeriod=25
Oct 30 18:56:59.257153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:59.256198    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded"
Oct 30 18:56:59.295456 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:59.295405    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:59.510304 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:59.510205    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:59.594262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:56:59.594216    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:00.099714 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:00.099213    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:00.313658 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:00.313615    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:00.339151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:00.328746    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:57:00.339151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:00.328805    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:05.132514 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.132474    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"]
Oct 30 18:57:05.133366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.133344    4372 cpu_manager.go:414] "containerMap: removing container" podUID="a68833d6-91b3-4c78-9115-aac5f70dbba8" containerName="main"
Oct 30 18:57:05.133366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.133366    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a68833d6-91b3-4c78-9115-aac5f70dbba8" containerName="main"
Oct 30 18:57:05.133542 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.133509    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="a68833d6-91b3-4c78-9115-aac5f70dbba8" containerName="main"
Oct 30 18:57:05.138080 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.138058    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"
Oct 30 18:57:05.152914 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.152891    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"]
Oct 30 18:57:05.153219 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.153158    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-493721\""
Oct 30 18:57:05.153326 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.153267    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-493721-dockercfg-q72z6\""
Oct 30 18:57:05.176090 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.176062    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"]
Oct 30 18:57:05.180977 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.180960    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"
Oct 30 18:57:05.190425 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.190403    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"]
Oct 30 18:57:05.315231 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.297207    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab9aaad1-b966-48c6-b2f1-88117497a527-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b\" (UID: \"ab9aaad1-b966-48c6-b2f1-88117497a527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"
Oct 30 18:57:05.335296 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.330911    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwbdj\" (UniqueName: \"kubernetes.io/projected/f62074d0-5307-4199-a143-ab7f53519248-kube-api-access-wwbdj\") pod \"payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns\" (UID: \"f62074d0-5307-4199-a143-ab7f53519248\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"
Oct 30 18:57:05.337253 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.337217    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9" podUID="e2f7d978-361f-4300-869e-985a9ff73cec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:57:05.358812 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.345942    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62074d0-5307-4199-a143-ab7f53519248-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns\" (UID: \"f62074d0-5307-4199-a143-ab7f53519248\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"
Oct 30 18:57:05.358812 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.352388    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bk5z\" (UniqueName: \"kubernetes.io/projected/ab9aaad1-b966-48c6-b2f1-88117497a527-kube-api-access-7bk5z\") pod \"payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b\" (UID: \"ab9aaad1-b966-48c6-b2f1-88117497a527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"
Oct 30 18:57:05.399356 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.399267    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54" podUID="136d8931-c397-4378-a8eb-c33842cf0776" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:57:05.465572 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.463348    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab9aaad1-b966-48c6-b2f1-88117497a527-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b\" (UID: \"ab9aaad1-b966-48c6-b2f1-88117497a527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"
Oct 30 18:57:05.465572 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.463420    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwbdj\" (UniqueName: \"kubernetes.io/projected/f62074d0-5307-4199-a143-ab7f53519248-kube-api-access-wwbdj\") pod \"payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns\" (UID: \"f62074d0-5307-4199-a143-ab7f53519248\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"
Oct 30 18:57:05.465572 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.463552    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62074d0-5307-4199-a143-ab7f53519248-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns\" (UID: \"f62074d0-5307-4199-a143-ab7f53519248\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"
Oct 30 18:57:05.465572 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.463651    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bk5z\" (UniqueName: \"kubernetes.io/projected/ab9aaad1-b966-48c6-b2f1-88117497a527-kube-api-access-7bk5z\") pod \"payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b\" (UID: \"ab9aaad1-b966-48c6-b2f1-88117497a527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"
Oct 30 18:57:05.468335 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.466003    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab9aaad1-b966-48c6-b2f1-88117497a527-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b\" (UID: \"ab9aaad1-b966-48c6-b2f1-88117497a527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"
Oct 30 18:57:05.469306 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.468985    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62074d0-5307-4199-a143-ab7f53519248-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns\" (UID: \"f62074d0-5307-4199-a143-ab7f53519248\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"
Oct 30 18:57:05.479434 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.479413    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwbdj\" (UniqueName: \"kubernetes.io/projected/f62074d0-5307-4199-a143-ab7f53519248-kube-api-access-wwbdj\") pod \"payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns\" (UID: \"f62074d0-5307-4199-a143-ab7f53519248\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"
Oct 30 18:57:05.480356 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.480341    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bk5z\" (UniqueName: \"kubernetes.io/projected/ab9aaad1-b966-48c6-b2f1-88117497a527-kube-api-access-7bk5z\") pod \"payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b\" (UID: \"ab9aaad1-b966-48c6-b2f1-88117497a527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"
Oct 30 18:57:05.505555 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.505533    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"
Oct 30 18:57:05.753010 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.752929    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"
Oct 30 18:57:05.794166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:05.791754    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"]
Oct 30 18:57:06.038234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:06.038211    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"]
Oct 30 18:57:06.039014 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:57:06.038990    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62074d0_5307_4199_a143_ab7f53519248.slice/crio-5d7cefd1f6ffe666f5d7e4b6a5b04dc0607849d19b0368e8f573e3ec8f868951 WatchSource:0}: Error finding container 5d7cefd1f6ffe666f5d7e4b6a5b04dc0607849d19b0368e8f573e3ec8f868951: Status 404 returned error can't find the container with id 5d7cefd1f6ffe666f5d7e4b6a5b04dc0607849d19b0368e8f573e3ec8f868951
Oct 30 18:57:06.370191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:06.370154    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b" event={"ID":"ab9aaad1-b966-48c6-b2f1-88117497a527","Type":"ContainerStarted","Data":"67a3848a844db5c51a8e7438899985a6cd3ae92e99724095b151ddf27e43d2af"}
Oct 30 18:57:06.370593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:06.370220    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"
Oct 30 18:57:06.370593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:06.370246    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b" event={"ID":"ab9aaad1-b966-48c6-b2f1-88117497a527","Type":"ContainerStarted","Data":"8d3653dcd975e9cda9941efec842de68c5330b706472ea3f6f44fb2e50597053"}
Oct 30 18:57:06.371687 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:06.371666    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns" event={"ID":"f62074d0-5307-4199-a143-ab7f53519248","Type":"ContainerStarted","Data":"294c8d82c7fe3403d66763531f17c3802c3c42de613875c937b4cda0b29395b1"}
Oct 30 18:57:06.371787 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:06.371704    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns" event={"ID":"f62074d0-5307-4199-a143-ab7f53519248","Type":"ContainerStarted","Data":"5d7cefd1f6ffe666f5d7e4b6a5b04dc0607849d19b0368e8f573e3ec8f868951"}
Oct 30 18:57:06.371787 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:06.371741    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"
Oct 30 18:57:06.392953 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:06.392903    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b" podStartSLOduration=1.39289079 podStartE2EDuration="1.39289079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:06.391357458 +0000 UTC m=+1440.259775789" watchObservedRunningTime="2026-10-30 18:57:06.39289079 +0000 UTC m=+1440.261309120"
Oct 30 18:57:06.413557 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:06.413517    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns" podStartSLOduration=1.413506028 podStartE2EDuration="1.413506028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:06.410342754 +0000 UTC m=+1440.278761085" watchObservedRunningTime="2026-10-30 18:57:06.413506028 +0000 UTC m=+1440.281924358"
Oct 30 18:57:12.156058 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.156021    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"]
Oct 30 18:57:12.161917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.161894    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:12.168116 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.168089    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-493721-serving-cert\""
Oct 30 18:57:12.185879 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.185855    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"]
Oct 30 18:57:12.343534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.343294    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-metrics-certs\") pod \"maas-api-e2e-ipp-b-493721-65765f864d-lgb9n\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:12.343534 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.343446    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbvhd\" (UniqueName: \"kubernetes.io/projected/90cac65e-3806-45fe-b407-c953c9a95560-kube-api-access-vbvhd\") pod \"maas-api-e2e-ipp-b-493721-65765f864d-lgb9n\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:12.343825 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.343569    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90cac65e-3806-45fe-b407-c953c9a95560-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-493721-65765f864d-lgb9n\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:12.343825 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.343650    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-maas-api-tls\") pod \"maas-api-e2e-ipp-b-493721-65765f864d-lgb9n\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:12.445011 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.444915    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-metrics-certs\") pod \"maas-api-e2e-ipp-b-493721-65765f864d-lgb9n\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:12.445011 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.444972    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbvhd\" (UniqueName: \"kubernetes.io/projected/90cac65e-3806-45fe-b407-c953c9a95560-kube-api-access-vbvhd\") pod \"maas-api-e2e-ipp-b-493721-65765f864d-lgb9n\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:12.445306 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.445087    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90cac65e-3806-45fe-b407-c953c9a95560-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-493721-65765f864d-lgb9n\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:12.445306 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.445209    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-maas-api-tls\") pod \"maas-api-e2e-ipp-b-493721-65765f864d-lgb9n\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:12.445419 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:57:12.445333    4372 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-serving-cert: secret "maas-api-e2e-ipp-b-493721-serving-cert" not found
Oct 30 18:57:12.445419 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:57:12.445404    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-maas-api-tls podName:90cac65e-3806-45fe-b407-c953c9a95560 nodeName:}" failed. No retries permitted until 2026-10-30 18:57:12.945389082 +0000 UTC m=+1446.813807394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-maas-api-tls") pod "maas-api-e2e-ipp-b-493721-65765f864d-lgb9n" (UID: "90cac65e-3806-45fe-b407-c953c9a95560") : secret "maas-api-e2e-ipp-b-493721-serving-cert" not found
Oct 30 18:57:12.445763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.445744    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90cac65e-3806-45fe-b407-c953c9a95560-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-493721-65765f864d-lgb9n\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:12.447695 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.447675    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-metrics-certs\") pod \"maas-api-e2e-ipp-b-493721-65765f864d-lgb9n\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:12.461925 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.461899    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbvhd\" (UniqueName: \"kubernetes.io/projected/90cac65e-3806-45fe-b407-c953c9a95560-kube-api-access-vbvhd\") pod \"maas-api-e2e-ipp-b-493721-65765f864d-lgb9n\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:12.961151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.960390    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-maas-api-tls\") pod \"maas-api-e2e-ipp-b-493721-65765f864d-lgb9n\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:12.964493 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:12.964452    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-maas-api-tls\") pod \"maas-api-e2e-ipp-b-493721-65765f864d-lgb9n\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:13.098191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:13.092618    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:13.449484 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:57:13.449447    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cac65e_3806_45fe_b407_c953c9a95560.slice/crio-258807eada44cd6e5adc39a4b6eb636516d514d16c404a13bb1dbc4967fa7162 WatchSource:0}: Error finding container 258807eada44cd6e5adc39a4b6eb636516d514d16c404a13bb1dbc4967fa7162: Status 404 returned error can't find the container with id 258807eada44cd6e5adc39a4b6eb636516d514d16c404a13bb1dbc4967fa7162
Oct 30 18:57:13.452395 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:13.452374    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"]
Oct 30 18:57:14.440350 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:14.440318    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n" event={"ID":"90cac65e-3806-45fe-b407-c953c9a95560","Type":"ContainerStarted","Data":"21e8d36c78303f5ea8c2dc9ddcf870caadd66a47db57aba83bf9a449fcb2d736"}
Oct 30 18:57:14.440527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:14.440362    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n" event={"ID":"90cac65e-3806-45fe-b407-c953c9a95560","Type":"ContainerStarted","Data":"258807eada44cd6e5adc39a4b6eb636516d514d16c404a13bb1dbc4967fa7162"}
Oct 30 18:57:14.440527 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:14.440449    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:15.214563 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:15.214520    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54" podUID="136d8931-c397-4378-a8eb-c33842cf0776" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:57:15.214984 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:15.214618    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9" podUID="e2f7d978-361f-4300-869e-985a9ff73cec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:57:17.381620 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:17.381594    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"
Oct 30 18:57:17.382040 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:17.381657    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"
Oct 30 18:57:17.405753 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:17.405702    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n" podStartSLOduration=4.933490636 podStartE2EDuration="5.405688873s" totalImagesPullingTime="472.198237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:14.514117772 +0000 UTC m=+1448.382536103" watchObservedRunningTime="2026-10-30 18:57:17.405688873 +0000 UTC m=+1451.274107257"
Oct 30 18:57:19.181837 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:57:19.181800    4372 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136d8931_c397_4378_a8eb_c33842cf0776.slice/crio-conmon-3de0c7b54a6948eb777485a6b5f946f9ed869763861349908edc8c023400138b.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 18:57:20.566463 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.566062    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"
Oct 30 18:57:20.623878 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.586621    4372 generic.go:356] "Generic (PLEG): container finished" podID="e2f7d978-361f-4300-869e-985a9ff73cec" containerID="dc1c6ce3b8f4bfe5768733417af250c61dcb251fdbfafd860d2619b58dca5c31" exitCode=0
Oct 30 18:57:20.640155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.638637    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": context deadline exceeded"
Oct 30 18:57:20.677155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.673681    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:20.681148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.680652    4372 generic.go:356] "Generic (PLEG): container finished" podID="136d8931-c397-4378-a8eb-c33842cf0776" containerID="3de0c7b54a6948eb777485a6b5f946f9ed869763861349908edc8c023400138b" exitCode=0
Oct 30 18:57:20.738203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.730368    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"
Oct 30 18:57:20.757526 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.757491    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:57:20.757797 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.757770    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9" event={"ID":"e2f7d978-361f-4300-869e-985a9ff73cec","Type":"ContainerDied","Data":"dc1c6ce3b8f4bfe5768733417af250c61dcb251fdbfafd860d2619b58dca5c31"}
Oct 30 18:57:20.757941 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.757925    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54" event={"ID":"136d8931-c397-4378-a8eb-c33842cf0776","Type":"ContainerDied","Data":"3de0c7b54a6948eb777485a6b5f946f9ed869763861349908edc8c023400138b"}
Oct 30 18:57:20.758089 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.758045    4372 scope.go:122] "RemoveContainer" containerID="dc1c6ce3b8f4bfe5768733417af250c61dcb251fdbfafd860d2619b58dca5c31"
Oct 30 18:57:20.836188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.758638    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136d8931-c397-4378-a8eb-c33842cf0776-kube-api-access-gtwll\" (UniqueName: \"kubernetes.io/projected/136d8931-c397-4378-a8eb-c33842cf0776-kube-api-access-gtwll\") pod \"136d8931-c397-4378-a8eb-c33842cf0776\" (UID: \"136d8931-c397-4378-a8eb-c33842cf0776\") "
Oct 30 18:57:20.836188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.758751    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f7d978-361f-4300-869e-985a9ff73cec-kube-api-access-c2vmm\" (UniqueName: \"kubernetes.io/projected/e2f7d978-361f-4300-869e-985a9ff73cec-kube-api-access-c2vmm\") pod \"e2f7d978-361f-4300-869e-985a9ff73cec\" (UID: \"e2f7d978-361f-4300-869e-985a9ff73cec\") "
Oct 30 18:57:20.836188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.758802    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f7d978-361f-4300-869e-985a9ff73cec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f7d978-361f-4300-869e-985a9ff73cec-plugins-config-volume\") pod \"e2f7d978-361f-4300-869e-985a9ff73cec\" (UID: \"e2f7d978-361f-4300-869e-985a9ff73cec\") "
Oct 30 18:57:20.836188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.758847    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136d8931-c397-4378-a8eb-c33842cf0776-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136d8931-c397-4378-a8eb-c33842cf0776-plugins-config-volume\") pod \"136d8931-c397-4378-a8eb-c33842cf0776\" (UID: \"136d8931-c397-4378-a8eb-c33842cf0776\") "
Oct 30 18:57:20.922421 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.922375    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136d8931-c397-4378-a8eb-c33842cf0776-plugins-config-volume" pod "136d8931-c397-4378-a8eb-c33842cf0776" (UID: "136d8931-c397-4378-a8eb-c33842cf0776"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:20.923735 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.923687    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f7d978-361f-4300-869e-985a9ff73cec-plugins-config-volume" pod "e2f7d978-361f-4300-869e-985a9ff73cec" (UID: "e2f7d978-361f-4300-869e-985a9ff73cec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:20.979421 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:20.979369    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f7d978-361f-4300-869e-985a9ff73cec-kube-api-access-c2vmm" pod "e2f7d978-361f-4300-869e-985a9ff73cec" (UID: "e2f7d978-361f-4300-869e-985a9ff73cec"). InnerVolumeSpecName "kube-api-access-c2vmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:21.038207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.009469    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f7d978-361f-4300-869e-985a9ff73cec-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:21.068652 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.068618    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136d8931-c397-4378-a8eb-c33842cf0776-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:21.068891 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.068875    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2vmm\" (UniqueName: \"kubernetes.io/projected/e2f7d978-361f-4300-869e-985a9ff73cec-kube-api-access-c2vmm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:21.092193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.017289    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136d8931-c397-4378-a8eb-c33842cf0776-kube-api-access-gtwll" pod "136d8931-c397-4378-a8eb-c33842cf0776" (UID: "136d8931-c397-4378-a8eb-c33842cf0776"). InnerVolumeSpecName "kube-api-access-gtwll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:21.247178 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.206248    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtwll\" (UniqueName: \"kubernetes.io/projected/136d8931-c397-4378-a8eb-c33842cf0776-kube-api-access-gtwll\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:21.300202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.268080    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:21.321478 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.293275    4372 scope.go:122] "RemoveContainer" containerID="3de0c7b54a6948eb777485a6b5f946f9ed869763861349908edc8c023400138b"
Oct 30 18:57:21.369301 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.369261    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 18:57:21.422157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.389655    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:21.422157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.409242    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:21.644148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.642204    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:21.666858 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.657838    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:21.671222 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.671196    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:57:21.676228 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.676198    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 18:57:21.678218 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.678186    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:57:21.692817 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.692797    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:57:21.703096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.703066    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54" event={"ID":"136d8931-c397-4378-a8eb-c33842cf0776","Type":"ContainerDied","Data":"7662019938d6710fc2f6952f2663a67aeac4f7f2fbfdf24bc71380fedf13c1c5"}
Oct 30 18:57:21.703523 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.703307    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"
Oct 30 18:57:21.710327 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.710302    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9" event={"ID":"e2f7d978-361f-4300-869e-985a9ff73cec","Type":"ContainerDied","Data":"f91c88a01fe2dde3d2e3a991486fdf2789b9e7a38ee287ea3fb5b2dea323fd85"}
Oct 30 18:57:21.710495 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.710251    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"
Oct 30 18:57:21.924205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:21.923217    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:57:22.355460 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:22.355195    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"]
Oct 30 18:57:22.358197 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:22.358166    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fa7b20-6df4b9cf6-2r7c9"]
Oct 30 18:57:22.369935 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:22.369895    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"]
Oct 30 18:57:22.374427 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:22.374401    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fa7b20-7875c49f67-skb54"]
Oct 30 18:57:24.311088 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:24.311036    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136d8931-c397-4378-a8eb-c33842cf0776" path="/var/lib/kubelet/pods/136d8931-c397-4378-a8eb-c33842cf0776/volumes"
Oct 30 18:57:24.311978 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:24.311952    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f7d978-361f-4300-869e-985a9ff73cec" path="/var/lib/kubelet/pods/e2f7d978-361f-4300-869e-985a9ff73cec/volumes"
Oct 30 18:57:26.789437 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:26.789403    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"]
Oct 30 18:57:27.069791 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.069711    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"]
Oct 30 18:57:27.070008 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.069970    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg" podUID="4fa388c0-7e72-4b86-b692-53d9c53ac30b" containerName="payload-processing" containerID="cri-o://0ac8cafa983c8ded200ba63bdb92da72ac59afa997aa2c9f97a79af04342feac" gracePeriod=30
Oct 30 18:57:27.093532 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.093501    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"]
Oct 30 18:57:27.094166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.094113    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b" podUID="71072275-42b7-4d5e-9e0c-cac53bf01426" containerName="payload-pre-processing" containerID="cri-o://b7d83cc279d11901d1bb2f7b632820a3dc4707d15fcbf1acdae4a601c4361dc8" gracePeriod=30
Oct 30 18:57:27.764658 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:57:27.764620    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:57:27.767296 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.767269    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"
Oct 30 18:57:27.773942 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.773916    4372 generic.go:356] "Generic (PLEG): container finished" podID="4fa388c0-7e72-4b86-b692-53d9c53ac30b" containerID="0ac8cafa983c8ded200ba63bdb92da72ac59afa997aa2c9f97a79af04342feac" exitCode=0
Oct 30 18:57:27.774080 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.773995    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"
Oct 30 18:57:27.774080 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.774034    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg" event={"ID":"4fa388c0-7e72-4b86-b692-53d9c53ac30b","Type":"ContainerDied","Data":"0ac8cafa983c8ded200ba63bdb92da72ac59afa997aa2c9f97a79af04342feac"}
Oct 30 18:57:27.774256 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.774080    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg" event={"ID":"4fa388c0-7e72-4b86-b692-53d9c53ac30b","Type":"ContainerDied","Data":"c890ae6bcee9f0a4a3a81e58ff8ca1097774ebca748b3e33afff4c8d4e3e30d6"}
Oct 30 18:57:27.774256 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.774110    4372 scope.go:122] "RemoveContainer" containerID="0ac8cafa983c8ded200ba63bdb92da72ac59afa997aa2c9f97a79af04342feac"
Oct 30 18:57:27.778624 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.778586    4372 generic.go:356] "Generic (PLEG): container finished" podID="71072275-42b7-4d5e-9e0c-cac53bf01426" containerID="b7d83cc279d11901d1bb2f7b632820a3dc4707d15fcbf1acdae4a601c4361dc8" exitCode=0
Oct 30 18:57:27.778761 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.778742    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b" event={"ID":"71072275-42b7-4d5e-9e0c-cac53bf01426","Type":"ContainerDied","Data":"b7d83cc279d11901d1bb2f7b632820a3dc4707d15fcbf1acdae4a601c4361dc8"}
Oct 30 18:57:27.803113 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.802071    4372 scope.go:122] "RemoveContainer" containerID="0ac8cafa983c8ded200ba63bdb92da72ac59afa997aa2c9f97a79af04342feac"
Oct 30 18:57:27.803501 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:57:27.803463    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac8cafa983c8ded200ba63bdb92da72ac59afa997aa2c9f97a79af04342feac\": container with ID starting with 0ac8cafa983c8ded200ba63bdb92da72ac59afa997aa2c9f97a79af04342feac not found: ID does not exist" containerID="0ac8cafa983c8ded200ba63bdb92da72ac59afa997aa2c9f97a79af04342feac"
Oct 30 18:57:27.803594 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.803516    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac8cafa983c8ded200ba63bdb92da72ac59afa997aa2c9f97a79af04342feac"} err="failed to get container status \"0ac8cafa983c8ded200ba63bdb92da72ac59afa997aa2c9f97a79af04342feac\": rpc error: code = NotFound desc = could not find container \"0ac8cafa983c8ded200ba63bdb92da72ac59afa997aa2c9f97a79af04342feac\": container with ID starting with 0ac8cafa983c8ded200ba63bdb92da72ac59afa997aa2c9f97a79af04342feac not found: ID does not exist"
Oct 30 18:57:27.804762 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.804736    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"
Oct 30 18:57:27.838086 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.836562    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fa388c0-7e72-4b86-b692-53d9c53ac30b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa388c0-7e72-4b86-b692-53d9c53ac30b-plugins-config-volume\") pod \"4fa388c0-7e72-4b86-b692-53d9c53ac30b\" (UID: \"4fa388c0-7e72-4b86-b692-53d9c53ac30b\") "
Oct 30 18:57:27.838086 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.836618    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa388c0-7e72-4b86-b692-53d9c53ac30b-kube-api-access-t79qr\" (UniqueName: \"kubernetes.io/projected/4fa388c0-7e72-4b86-b692-53d9c53ac30b-kube-api-access-t79qr\") pod \"4fa388c0-7e72-4b86-b692-53d9c53ac30b\" (UID: \"4fa388c0-7e72-4b86-b692-53d9c53ac30b\") "
Oct 30 18:57:27.838086 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.836886    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa388c0-7e72-4b86-b692-53d9c53ac30b-plugins-config-volume" pod "4fa388c0-7e72-4b86-b692-53d9c53ac30b" (UID: "4fa388c0-7e72-4b86-b692-53d9c53ac30b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:27.838086 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.837417    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa388c0-7e72-4b86-b692-53d9c53ac30b-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:27.839654 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.839623    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa388c0-7e72-4b86-b692-53d9c53ac30b-kube-api-access-t79qr" pod "4fa388c0-7e72-4b86-b692-53d9c53ac30b" (UID: "4fa388c0-7e72-4b86-b692-53d9c53ac30b"). InnerVolumeSpecName "kube-api-access-t79qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:27.938118 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.938089    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71072275-42b7-4d5e-9e0c-cac53bf01426-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71072275-42b7-4d5e-9e0c-cac53bf01426-plugins-config-volume\") pod \"71072275-42b7-4d5e-9e0c-cac53bf01426\" (UID: \"71072275-42b7-4d5e-9e0c-cac53bf01426\") "
Oct 30 18:57:27.938270 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.938152    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71072275-42b7-4d5e-9e0c-cac53bf01426-kube-api-access-m6m9w\" (UniqueName: \"kubernetes.io/projected/71072275-42b7-4d5e-9e0c-cac53bf01426-kube-api-access-m6m9w\") pod \"71072275-42b7-4d5e-9e0c-cac53bf01426\" (UID: \"71072275-42b7-4d5e-9e0c-cac53bf01426\") "
Oct 30 18:57:27.938439 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.938417    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71072275-42b7-4d5e-9e0c-cac53bf01426-plugins-config-volume" pod "71072275-42b7-4d5e-9e0c-cac53bf01426" (UID: "71072275-42b7-4d5e-9e0c-cac53bf01426"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:27.938698 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.938680    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t79qr\" (UniqueName: \"kubernetes.io/projected/4fa388c0-7e72-4b86-b692-53d9c53ac30b-kube-api-access-t79qr\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:27.938698 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.938697    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71072275-42b7-4d5e-9e0c-cac53bf01426-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:27.940535 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:27.940513    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71072275-42b7-4d5e-9e0c-cac53bf01426-kube-api-access-m6m9w" pod "71072275-42b7-4d5e-9e0c-cac53bf01426" (UID: "71072275-42b7-4d5e-9e0c-cac53bf01426"). InnerVolumeSpecName "kube-api-access-m6m9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:28.040570 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:28.040548    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6m9w\" (UniqueName: \"kubernetes.io/projected/71072275-42b7-4d5e-9e0c-cac53bf01426-kube-api-access-m6m9w\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:28.099597 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:28.099539    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"]
Oct 30 18:57:28.104285 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:28.104265    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-348877-779c9dc8f4-crndg"]
Oct 30 18:57:28.265917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:28.265893    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fa388c0-7e72-4b86-b692-53d9c53ac30b" path="/var/lib/kubelet/pods/4fa388c0-7e72-4b86-b692-53d9c53ac30b/volumes"
Oct 30 18:57:28.788962 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:28.788933    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"
Oct 30 18:57:28.789185 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:28.788971    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b" event={"ID":"71072275-42b7-4d5e-9e0c-cac53bf01426","Type":"ContainerDied","Data":"fae602f2e7a4e3dca8cc9da9c2a45158c8861c3fdc3c8aabcff4b2f99b42b41a"}
Oct 30 18:57:28.789185 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:28.789030    4372 scope.go:122] "RemoveContainer" containerID="b7d83cc279d11901d1bb2f7b632820a3dc4707d15fcbf1acdae4a601c4361dc8"
Oct 30 18:57:28.809924 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:28.809897    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"]
Oct 30 18:57:28.815006 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:28.814985    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-348877-7c85fbd7f-zpl2b"]
Oct 30 18:57:29.051955 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:29.051861    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"]
Oct 30 18:57:29.081022 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:29.079960    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"]
Oct 30 18:57:30.266743 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:30.266706    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71072275-42b7-4d5e-9e0c-cac53bf01426" path="/var/lib/kubelet/pods/71072275-42b7-4d5e-9e0c-cac53bf01426/volumes"
Oct 30 18:57:32.775561 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:32.775530    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:57:34.055293 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:34.055215    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b" podUID="ab9aaad1-b966-48c6-b2f1-88117497a527" containerName="payload-processing" containerID="cri-o://67a3848a844db5c51a8e7438899985a6cd3ae92e99724095b151ddf27e43d2af" gracePeriod=25
Oct 30 18:57:34.099465 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:34.091556    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns" podUID="f62074d0-5307-4199-a143-ab7f53519248" containerName="payload-pre-processing" containerID="cri-o://294c8d82c7fe3403d66763531f17c3802c3c42de613875c937b4cda0b29395b1" gracePeriod=25
Oct 30 18:57:34.882992 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:34.882940    4372 generic.go:356] "Generic (PLEG): container finished" podID="f62074d0-5307-4199-a143-ab7f53519248" containerID="294c8d82c7fe3403d66763531f17c3802c3c42de613875c937b4cda0b29395b1" exitCode=0
Oct 30 18:57:34.883548 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:34.883500    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns" event={"ID":"f62074d0-5307-4199-a143-ab7f53519248","Type":"ContainerDied","Data":"294c8d82c7fe3403d66763531f17c3802c3c42de613875c937b4cda0b29395b1"}
Oct 30 18:57:34.886049 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:34.885913    4372 generic.go:356] "Generic (PLEG): container finished" podID="ab9aaad1-b966-48c6-b2f1-88117497a527" containerID="67a3848a844db5c51a8e7438899985a6cd3ae92e99724095b151ddf27e43d2af" exitCode=0
Oct 30 18:57:34.886049 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:34.885969    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b" event={"ID":"ab9aaad1-b966-48c6-b2f1-88117497a527","Type":"ContainerDied","Data":"67a3848a844db5c51a8e7438899985a6cd3ae92e99724095b151ddf27e43d2af"}
Oct 30 18:57:35.208211 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:35.207336    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"
Oct 30 18:57:35.294153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:35.252962    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62074d0-5307-4199-a143-ab7f53519248-kube-api-access-wwbdj\" (UniqueName: \"kubernetes.io/projected/f62074d0-5307-4199-a143-ab7f53519248-kube-api-access-wwbdj\") pod \"f62074d0-5307-4199-a143-ab7f53519248\" (UID: \"f62074d0-5307-4199-a143-ab7f53519248\") "
Oct 30 18:57:35.294153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:35.253093    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f62074d0-5307-4199-a143-ab7f53519248-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62074d0-5307-4199-a143-ab7f53519248-plugins-config-volume\") pod \"f62074d0-5307-4199-a143-ab7f53519248\" (UID: \"f62074d0-5307-4199-a143-ab7f53519248\") "
Oct 30 18:57:35.294153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:35.255350    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62074d0-5307-4199-a143-ab7f53519248-plugins-config-volume" pod "f62074d0-5307-4199-a143-ab7f53519248" (UID: "f62074d0-5307-4199-a143-ab7f53519248"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:35.503631 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:35.503562    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62074d0-5307-4199-a143-ab7f53519248-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:35.611333 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:35.556176    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62074d0-5307-4199-a143-ab7f53519248-kube-api-access-wwbdj" pod "f62074d0-5307-4199-a143-ab7f53519248" (UID: "f62074d0-5307-4199-a143-ab7f53519248"). InnerVolumeSpecName "kube-api-access-wwbdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:35.611333 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:35.556651    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:35.639918 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:35.613359    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwbdj\" (UniqueName: \"kubernetes.io/projected/f62074d0-5307-4199-a143-ab7f53519248-kube-api-access-wwbdj\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:35.769020 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:35.767629    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"
Oct 30 18:57:35.840013 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:35.839972    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:35.849275 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:35.849236    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 30 18:57:35.873232 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:35.865375    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab9aaad1-b966-48c6-b2f1-88117497a527-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab9aaad1-b966-48c6-b2f1-88117497a527-plugins-config-volume\") pod \"ab9aaad1-b966-48c6-b2f1-88117497a527\" (UID: \"ab9aaad1-b966-48c6-b2f1-88117497a527\") "
Oct 30 18:57:35.884419 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:35.884380    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab9aaad1-b966-48c6-b2f1-88117497a527-plugins-config-volume" pod "ab9aaad1-b966-48c6-b2f1-88117497a527" (UID: "ab9aaad1-b966-48c6-b2f1-88117497a527"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:35.989922 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:35.989879    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9aaad1-b966-48c6-b2f1-88117497a527-kube-api-access-7bk5z\" (UniqueName: \"kubernetes.io/projected/ab9aaad1-b966-48c6-b2f1-88117497a527-kube-api-access-7bk5z\") pod \"ab9aaad1-b966-48c6-b2f1-88117497a527\" (UID: \"ab9aaad1-b966-48c6-b2f1-88117497a527\") "
Oct 30 18:57:35.997767 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:35.997735    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab9aaad1-b966-48c6-b2f1-88117497a527-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:36.046486 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.044223    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:36.051149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.048108    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 18:57:36.377149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.368332    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:36.390746 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.390705    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9aaad1-b966-48c6-b2f1-88117497a527-kube-api-access-7bk5z" pod "ab9aaad1-b966-48c6-b2f1-88117497a527" (UID: "ab9aaad1-b966-48c6-b2f1-88117497a527"). InnerVolumeSpecName "kube-api-access-7bk5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:36.497162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.493598    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bk5z\" (UniqueName: \"kubernetes.io/projected/ab9aaad1-b966-48c6-b2f1-88117497a527-kube-api-access-7bk5z\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:36.564073 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.563992    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"
Oct 30 18:57:36.569208 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.567548    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"
Oct 30 18:57:36.570192 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.570143    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns" event={"ID":"f62074d0-5307-4199-a143-ab7f53519248","Type":"ContainerDied","Data":"5d7cefd1f6ffe666f5d7e4b6a5b04dc0607849d19b0368e8f573e3ec8f868951"}
Oct 30 18:57:36.570306 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.570211    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"]
Oct 30 18:57:36.570306 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.570254    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b" event={"ID":"ab9aaad1-b966-48c6-b2f1-88117497a527","Type":"ContainerDied","Data":"8d3653dcd975e9cda9941efec842de68c5330b706472ea3f6f44fb2e50597053"}
Oct 30 18:57:36.570463 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.570437    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74" podUID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" containerName="istio-proxy" containerID="cri-o://e6ba8b07a6fe0acca60eeedc43c912190fd89c1b21b5e3e1abb5467dc280801d" gracePeriod=30
Oct 30 18:57:36.575979 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.575945    4372 scope.go:122] "RemoveContainer" containerID="294c8d82c7fe3403d66763531f17c3802c3c42de613875c937b4cda0b29395b1"
Oct 30 18:57:36.688787 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.688759    4372 scope.go:122] "RemoveContainer" containerID="67a3848a844db5c51a8e7438899985a6cd3ae92e99724095b151ddf27e43d2af"
Oct 30 18:57:36.864500 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.864253    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": context deadline exceeded"
Oct 30 18:57:36.865116 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.865093    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:57:36.928267 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.928191    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"]
Oct 30 18:57:36.949890 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.949574    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:57:36.950579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.950534    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-493721-bff84cff8-6qr5b"]
Oct 30 18:57:36.962170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.961226    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"]
Oct 30 18:57:36.967053 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:36.967030    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-6fbb995b95-vlbns"]
Oct 30 18:57:37.003684 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.003643    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"]
Oct 30 18:57:37.023316 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.004746    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f62074d0-5307-4199-a143-ab7f53519248" containerName="payload-pre-processing"
Oct 30 18:57:37.132151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.114508    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62074d0-5307-4199-a143-ab7f53519248" containerName="payload-pre-processing"
Oct 30 18:57:37.133311 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.133284    4372 cpu_manager.go:414] "containerMap: removing container" podUID="ab9aaad1-b966-48c6-b2f1-88117497a527" containerName="payload-processing"
Oct 30 18:57:37.133621 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.133604    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab9aaad1-b966-48c6-b2f1-88117497a527" containerName="payload-processing"
Oct 30 18:57:37.137167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.133833    4372 cpu_manager.go:414] "containerMap: removing container" podUID="4fa388c0-7e72-4b86-b692-53d9c53ac30b" containerName="payload-processing"
Oct 30 18:57:37.137341 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.137326    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa388c0-7e72-4b86-b692-53d9c53ac30b" containerName="payload-processing"
Oct 30 18:57:37.141216 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.141194    4372 cpu_manager.go:414] "containerMap: removing container" podUID="71072275-42b7-4d5e-9e0c-cac53bf01426" containerName="payload-pre-processing"
Oct 30 18:57:37.141377 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.141361    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71072275-42b7-4d5e-9e0c-cac53bf01426" containerName="payload-pre-processing"
Oct 30 18:57:37.145206 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.145182    4372 cpu_manager.go:414] "containerMap: removing container" podUID="136d8931-c397-4378-a8eb-c33842cf0776" containerName="payload-pre-processing"
Oct 30 18:57:37.160607 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.160544    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136d8931-c397-4378-a8eb-c33842cf0776" containerName="payload-pre-processing"
Oct 30 18:57:37.160841 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.160826    4372 cpu_manager.go:414] "containerMap: removing container" podUID="e2f7d978-361f-4300-869e-985a9ff73cec" containerName="payload-processing"
Oct 30 18:57:37.160952 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.160941    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f7d978-361f-4300-869e-985a9ff73cec" containerName="payload-processing"
Oct 30 18:57:37.161347 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.161318    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f7d978-361f-4300-869e-985a9ff73cec" containerName="payload-processing"
Oct 30 18:57:37.161487 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.161477    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62074d0-5307-4199-a143-ab7f53519248" containerName="payload-pre-processing"
Oct 30 18:57:37.169166 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.161573    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab9aaad1-b966-48c6-b2f1-88117497a527" containerName="payload-processing"
Oct 30 18:57:37.169395 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.169365    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa388c0-7e72-4b86-b692-53d9c53ac30b" containerName="payload-processing"
Oct 30 18:57:37.169533 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.169521    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="71072275-42b7-4d5e-9e0c-cac53bf01426" containerName="payload-pre-processing"
Oct 30 18:57:37.169647 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.169636    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="136d8931-c397-4378-a8eb-c33842cf0776" containerName="payload-pre-processing"
Oct 30 18:57:37.229305 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.229212    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"
Oct 30 18:57:37.272462 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.270341    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-493721\""
Oct 30 18:57:37.286489 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.286459    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-493721-dockercfg-bk48r\""
Oct 30 18:57:37.321154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.319280    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"]
Oct 30 18:57:37.321154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.320328    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jlgg\" (UniqueName: \"kubernetes.io/projected/8db96541-6147-4197-8623-767675ace383-kube-api-access-2jlgg\") pod \"payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v\" (UID: \"8db96541-6147-4197-8623-767675ace383\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"
Oct 30 18:57:37.321154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.320390    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db96541-6147-4197-8623-767675ace383-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v\" (UID: \"8db96541-6147-4197-8623-767675ace383\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"
Oct 30 18:57:37.328346 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.328321    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"]
Oct 30 18:57:37.358619 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.358589    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"]
Oct 30 18:57:37.358980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.358958    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"
Oct 30 18:57:37.423843 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.423800    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20da1af7-a9e4-45e0-b78e-712910be1c68-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc\" (UID: \"20da1af7-a9e4-45e0-b78e-712910be1c68\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"
Oct 30 18:57:37.424353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.423863    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db96541-6147-4197-8623-767675ace383-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v\" (UID: \"8db96541-6147-4197-8623-767675ace383\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"
Oct 30 18:57:37.425357 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.425328    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g7kt\" (UniqueName: \"kubernetes.io/projected/20da1af7-a9e4-45e0-b78e-712910be1c68-kube-api-access-2g7kt\") pod \"payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc\" (UID: \"20da1af7-a9e4-45e0-b78e-712910be1c68\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"
Oct 30 18:57:37.425520 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.425504    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jlgg\" (UniqueName: \"kubernetes.io/projected/8db96541-6147-4197-8623-767675ace383-kube-api-access-2jlgg\") pod \"payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v\" (UID: \"8db96541-6147-4197-8623-767675ace383\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"
Oct 30 18:57:37.426107 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.426079    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db96541-6147-4197-8623-767675ace383-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v\" (UID: \"8db96541-6147-4197-8623-767675ace383\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"
Oct 30 18:57:37.443288 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.440363    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jlgg\" (UniqueName: \"kubernetes.io/projected/8db96541-6147-4197-8623-767675ace383-kube-api-access-2jlgg\") pod \"payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v\" (UID: \"8db96541-6147-4197-8623-767675ace383\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"
Oct 30 18:57:37.527226 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.527120    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g7kt\" (UniqueName: \"kubernetes.io/projected/20da1af7-a9e4-45e0-b78e-712910be1c68-kube-api-access-2g7kt\") pod \"payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc\" (UID: \"20da1af7-a9e4-45e0-b78e-712910be1c68\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"
Oct 30 18:57:37.527226 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.527201    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20da1af7-a9e4-45e0-b78e-712910be1c68-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc\" (UID: \"20da1af7-a9e4-45e0-b78e-712910be1c68\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"
Oct 30 18:57:37.527917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.527898    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20da1af7-a9e4-45e0-b78e-712910be1c68-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc\" (UID: \"20da1af7-a9e4-45e0-b78e-712910be1c68\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"
Oct 30 18:57:37.543792 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.543765    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g7kt\" (UniqueName: \"kubernetes.io/projected/20da1af7-a9e4-45e0-b78e-712910be1c68-kube-api-access-2g7kt\") pod \"payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc\" (UID: \"20da1af7-a9e4-45e0-b78e-712910be1c68\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"
Oct 30 18:57:37.612356 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.612314    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"
Oct 30 18:57:37.722193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.721438    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"
Oct 30 18:57:37.907573 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:37.907540    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"]
Oct 30 18:57:37.907834 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:57:37.907811    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db96541_6147_4197_8623_767675ace383.slice/crio-6a98876dbd13ff8199e2467c07265030b8d9d9ea07ede1602488135ecf0eda28 WatchSource:0}: Error finding container 6a98876dbd13ff8199e2467c07265030b8d9d9ea07ede1602488135ecf0eda28: Status 404 returned error can't find the container with id 6a98876dbd13ff8199e2467c07265030b8d9d9ea07ede1602488135ecf0eda28
Oct 30 18:57:38.020781 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:57:38.020748    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20da1af7_a9e4_45e0_b78e_712910be1c68.slice/crio-959cb65b50d80d00557fe44d1d71ee08cb64b52893138d6e6871c92391dbcf82 WatchSource:0}: Error finding container 959cb65b50d80d00557fe44d1d71ee08cb64b52893138d6e6871c92391dbcf82: Status 404 returned error can't find the container with id 959cb65b50d80d00557fe44d1d71ee08cb64b52893138d6e6871c92391dbcf82
Oct 30 18:57:38.021665 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:38.021639    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"]
Oct 30 18:57:38.281868 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:38.281824    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab9aaad1-b966-48c6-b2f1-88117497a527" path="/var/lib/kubelet/pods/ab9aaad1-b966-48c6-b2f1-88117497a527/volumes"
Oct 30 18:57:38.282405 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:38.282388    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f62074d0-5307-4199-a143-ab7f53519248" path="/var/lib/kubelet/pods/f62074d0-5307-4199-a143-ab7f53519248/volumes"
Oct 30 18:57:38.322296 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:38.322219    4372 patch_prober.go:29] interesting pod/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:57:38.322296 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:38.322279    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74" podUID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:57:38.595634 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:38.594890    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc" event={"ID":"20da1af7-a9e4-45e0-b78e-712910be1c68","Type":"ContainerStarted","Data":"a41cebc3e6de9412bd10e7e8e6fb995ad3c9ae2fc635917a8709d05e4e471db3"}
Oct 30 18:57:38.595634 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:38.595075    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc" event={"ID":"20da1af7-a9e4-45e0-b78e-712910be1c68","Type":"ContainerStarted","Data":"959cb65b50d80d00557fe44d1d71ee08cb64b52893138d6e6871c92391dbcf82"}
Oct 30 18:57:38.596358 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:38.596335    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"
Oct 30 18:57:38.605732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:38.604726    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v" event={"ID":"8db96541-6147-4197-8623-767675ace383","Type":"ContainerStarted","Data":"602e4d3f0ab202eb8ed5cb438232b6edd27cc4dbacf4b46e5dab4937c990c647"}
Oct 30 18:57:38.605732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:38.604785    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v" event={"ID":"8db96541-6147-4197-8623-767675ace383","Type":"ContainerStarted","Data":"6a98876dbd13ff8199e2467c07265030b8d9d9ea07ede1602488135ecf0eda28"}
Oct 30 18:57:38.605732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:38.605656    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"
Oct 30 18:57:38.632439 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:38.632378    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc" podStartSLOduration=2.632359194 podStartE2EDuration="2.632359194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:38.628542356 +0000 UTC m=+1472.496960687" watchObservedRunningTime="2026-10-30 18:57:38.632359194 +0000 UTC m=+1472.500777526"
Oct 30 18:57:38.657476 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:38.657415    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v" podStartSLOduration=2.657393586 podStartE2EDuration="2.657393586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:38.651775016 +0000 UTC m=+1472.520193348" watchObservedRunningTime="2026-10-30 18:57:38.657393586 +0000 UTC m=+1472.525811914"
Oct 30 18:57:41.811620 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.811578    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerName="main" containerID="cri-o://11749201dd156a1bb61e77d95e7d646c690cfb775380a42261d8ce690ab8c7f7" gracePeriod=45
Oct 30 18:57:41.904810 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.904747    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:57:41.937109 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.937076    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-workload-certs\") pod \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") "
Oct 30 18:57:41.937298 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.937216    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-podinfo\") pod \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") "
Oct 30 18:57:41.937371 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.937348    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-kube-api-access-nlgfl\" (UniqueName: \"kubernetes.io/projected/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-kube-api-access-nlgfl\") pod \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") "
Oct 30 18:57:41.937764 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.937433    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-data\") pod \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") "
Oct 30 18:57:41.937764 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.937498    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-workload-socket\") pod \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") "
Oct 30 18:57:41.937764 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.937540    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-credential-socket\") pod \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") "
Oct 30 18:57:41.937764 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.937613    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-token\" (UniqueName: \"kubernetes.io/projected/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-token\") pod \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") "
Oct 30 18:57:41.937764 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.937656    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-envoy\") pod \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") "
Oct 30 18:57:41.937764 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.937702    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-workload-certs" pod "57b527cf-ac31-44b7-8ff0-6e40ee60ce32" (UID: "57b527cf-ac31-44b7-8ff0-6e40ee60ce32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:41.937764 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.937731    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istiod-ca-cert\") pod \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\" (UID: \"57b527cf-ac31-44b7-8ff0-6e40ee60ce32\") "
Oct 30 18:57:41.938208 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.937885    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-credential-socket" pod "57b527cf-ac31-44b7-8ff0-6e40ee60ce32" (UID: "57b527cf-ac31-44b7-8ff0-6e40ee60ce32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:41.938208 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.938056    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-data" pod "57b527cf-ac31-44b7-8ff0-6e40ee60ce32" (UID: "57b527cf-ac31-44b7-8ff0-6e40ee60ce32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:41.938433 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.938408    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istiod-ca-cert" pod "57b527cf-ac31-44b7-8ff0-6e40ee60ce32" (UID: "57b527cf-ac31-44b7-8ff0-6e40ee60ce32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:41.938506 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.938443    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:41.938506 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.938466    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:41.938506 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.938481    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:41.938673 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.938562    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-workload-socket" pod "57b527cf-ac31-44b7-8ff0-6e40ee60ce32" (UID: "57b527cf-ac31-44b7-8ff0-6e40ee60ce32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:41.941154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.941100    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-kube-api-access-nlgfl" pod "57b527cf-ac31-44b7-8ff0-6e40ee60ce32" (UID: "57b527cf-ac31-44b7-8ff0-6e40ee60ce32"). InnerVolumeSpecName "kube-api-access-nlgfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:41.941374 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.941290    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-podinfo" pod "57b527cf-ac31-44b7-8ff0-6e40ee60ce32" (UID: "57b527cf-ac31-44b7-8ff0-6e40ee60ce32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:57:41.944023 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.944000    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-token" pod "57b527cf-ac31-44b7-8ff0-6e40ee60ce32" (UID: "57b527cf-ac31-44b7-8ff0-6e40ee60ce32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:41.944148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:41.944097    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-envoy" pod "57b527cf-ac31-44b7-8ff0-6e40ee60ce32" (UID: "57b527cf-ac31-44b7-8ff0-6e40ee60ce32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:42.046224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.045323    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:42.046224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.045360    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:42.046224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.045375    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:42.046224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.045390    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:42.046224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.045403    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlgfl\" (UniqueName: \"kubernetes.io/projected/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-kube-api-access-nlgfl\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:42.046224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.045417    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b527cf-ac31-44b7-8ff0-6e40ee60ce32-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:42.160271 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.160217    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:57:42.248221 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.248195    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-tmp-dir\") pod \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") "
Oct 30 18:57:42.248390 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.248240    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-tls-certs\") pod \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") "
Oct 30 18:57:42.248390 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.248300    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-kube-api-access-wrltq\" (UniqueName: \"kubernetes.io/projected/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-kube-api-access-wrltq\") pod \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") "
Oct 30 18:57:42.248390 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.248316    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-dshm\") pod \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") "
Oct 30 18:57:42.248579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.248464    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-model-cache\") pod \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") "
Oct 30 18:57:42.248579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.248463    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-tmp-dir" pod "0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" (UID: "0ae2b494-2225-49bb-a1a5-2e45bc8cc12d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:42.248579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.248527    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-home\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-home\") pod \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\" (UID: \"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d\") "
Oct 30 18:57:42.248756 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.248669    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-model-cache" pod "0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" (UID: "0ae2b494-2225-49bb-a1a5-2e45bc8cc12d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:42.248756 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.248698    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-home" pod "0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" (UID: "0ae2b494-2225-49bb-a1a5-2e45bc8cc12d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:42.249163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.249143    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:42.249263 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.249164    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:42.249263 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.249177    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:42.250717 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.250695    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-dshm" pod "0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" (UID: "0ae2b494-2225-49bb-a1a5-2e45bc8cc12d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:42.251114 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.251090    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-tls-certs" pod "0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" (UID: "0ae2b494-2225-49bb-a1a5-2e45bc8cc12d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:42.251221 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.251118    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-kube-api-access-wrltq" pod "0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" (UID: "0ae2b494-2225-49bb-a1a5-2e45bc8cc12d"). InnerVolumeSpecName "kube-api-access-wrltq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:42.359141 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.351061    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrltq\" (UniqueName: \"kubernetes.io/projected/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-kube-api-access-wrltq\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:42.359141 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.351170    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:42.359141 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.351200    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:42.635074 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.635045    4372 generic.go:356] "Generic (PLEG): container finished" podID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerID="11749201dd156a1bb61e77d95e7d646c690cfb775380a42261d8ce690ab8c7f7" exitCode=0
Oct 30 18:57:42.635280 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.635106    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"
Oct 30 18:57:42.635280 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.635142    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" event={"ID":"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d","Type":"ContainerDied","Data":"11749201dd156a1bb61e77d95e7d646c690cfb775380a42261d8ce690ab8c7f7"}
Oct 30 18:57:42.635280 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.635187    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn" event={"ID":"0ae2b494-2225-49bb-a1a5-2e45bc8cc12d","Type":"ContainerDied","Data":"31a99fe44e722acfc98de205d3a78bda3715368c3f3846fba698bafce6750b26"}
Oct 30 18:57:42.635280 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.635210    4372 scope.go:122] "RemoveContainer" containerID="11749201dd156a1bb61e77d95e7d646c690cfb775380a42261d8ce690ab8c7f7"
Oct 30 18:57:42.637569 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.637527    4372 generic.go:356] "Generic (PLEG): container finished" podID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" containerID="e6ba8b07a6fe0acca60eeedc43c912190fd89c1b21b5e3e1abb5467dc280801d" exitCode=0
Oct 30 18:57:42.637703 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.637604    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74" event={"ID":"57b527cf-ac31-44b7-8ff0-6e40ee60ce32","Type":"ContainerDied","Data":"e6ba8b07a6fe0acca60eeedc43c912190fd89c1b21b5e3e1abb5467dc280801d"}
Oct 30 18:57:42.637703 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.637624    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"
Oct 30 18:57:42.637703 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.637683    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74" event={"ID":"57b527cf-ac31-44b7-8ff0-6e40ee60ce32","Type":"ContainerDied","Data":"a1121c73adfa343b1d5d69b1cae4de4fbe806ed0311768907bda0125872e1497"}
Oct 30 18:57:42.647095 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.647074    4372 scope.go:122] "RemoveContainer" containerID="11749201dd156a1bb61e77d95e7d646c690cfb775380a42261d8ce690ab8c7f7"
Oct 30 18:57:42.647380 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:57:42.647346    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11749201dd156a1bb61e77d95e7d646c690cfb775380a42261d8ce690ab8c7f7\": container with ID starting with 11749201dd156a1bb61e77d95e7d646c690cfb775380a42261d8ce690ab8c7f7 not found: ID does not exist" containerID="11749201dd156a1bb61e77d95e7d646c690cfb775380a42261d8ce690ab8c7f7"
Oct 30 18:57:42.647429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.647385    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11749201dd156a1bb61e77d95e7d646c690cfb775380a42261d8ce690ab8c7f7"} err="failed to get container status \"11749201dd156a1bb61e77d95e7d646c690cfb775380a42261d8ce690ab8c7f7\": rpc error: code = NotFound desc = could not find container \"11749201dd156a1bb61e77d95e7d646c690cfb775380a42261d8ce690ab8c7f7\": container with ID starting with 11749201dd156a1bb61e77d95e7d646c690cfb775380a42261d8ce690ab8c7f7 not found: ID does not exist"
Oct 30 18:57:42.647429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.647402    4372 scope.go:122] "RemoveContainer" containerID="e6ba8b07a6fe0acca60eeedc43c912190fd89c1b21b5e3e1abb5467dc280801d"
Oct 30 18:57:42.656622 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.656603    4372 scope.go:122] "RemoveContainer" containerID="ad245b65e6a015a18a9ba10eafe03212a0977da46bee1264b343f742b4563490"
Oct 30 18:57:42.663078 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.663058    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"]
Oct 30 18:57:42.668113 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.668096    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-fa7b20/praxis-route-fa7b20-kserve-67df94c59c-fctbn"]
Oct 30 18:57:42.676001 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.675982    4372 scope.go:122] "RemoveContainer" containerID="e6ba8b07a6fe0acca60eeedc43c912190fd89c1b21b5e3e1abb5467dc280801d"
Oct 30 18:57:42.676282 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:57:42.676265    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ba8b07a6fe0acca60eeedc43c912190fd89c1b21b5e3e1abb5467dc280801d\": container with ID starting with e6ba8b07a6fe0acca60eeedc43c912190fd89c1b21b5e3e1abb5467dc280801d not found: ID does not exist" containerID="e6ba8b07a6fe0acca60eeedc43c912190fd89c1b21b5e3e1abb5467dc280801d"
Oct 30 18:57:42.676334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.676287    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ba8b07a6fe0acca60eeedc43c912190fd89c1b21b5e3e1abb5467dc280801d"} err="failed to get container status \"e6ba8b07a6fe0acca60eeedc43c912190fd89c1b21b5e3e1abb5467dc280801d\": rpc error: code = NotFound desc = could not find container \"e6ba8b07a6fe0acca60eeedc43c912190fd89c1b21b5e3e1abb5467dc280801d\": container with ID starting with e6ba8b07a6fe0acca60eeedc43c912190fd89c1b21b5e3e1abb5467dc280801d not found: ID does not exist"
Oct 30 18:57:42.676334 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.676302    4372 scope.go:122] "RemoveContainer" containerID="ad245b65e6a015a18a9ba10eafe03212a0977da46bee1264b343f742b4563490"
Oct 30 18:57:42.676552 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:57:42.676533    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad245b65e6a015a18a9ba10eafe03212a0977da46bee1264b343f742b4563490\": container with ID starting with ad245b65e6a015a18a9ba10eafe03212a0977da46bee1264b343f742b4563490 not found: ID does not exist" containerID="ad245b65e6a015a18a9ba10eafe03212a0977da46bee1264b343f742b4563490"
Oct 30 18:57:42.676602 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.676560    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad245b65e6a015a18a9ba10eafe03212a0977da46bee1264b343f742b4563490"} err="failed to get container status \"ad245b65e6a015a18a9ba10eafe03212a0977da46bee1264b343f742b4563490\": rpc error: code = NotFound desc = could not find container \"ad245b65e6a015a18a9ba10eafe03212a0977da46bee1264b343f742b4563490\": container with ID starting with ad245b65e6a015a18a9ba10eafe03212a0977da46bee1264b343f742b4563490 not found: ID does not exist"
Oct 30 18:57:42.727621 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.727599    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"]
Oct 30 18:57:42.737399 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:42.737374    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-348877-openshift-default-5567478d87-mkj74"]
Oct 30 18:57:43.901031 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.900998    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"]
Oct 30 18:57:43.901579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.901565    4372 cpu_manager.go:414] "containerMap: removing container" podUID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" containerName="istio-proxy"
Oct 30 18:57:43.901633 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.901582    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" containerName="istio-proxy"
Oct 30 18:57:43.901633 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.901605    4372 cpu_manager.go:414] "containerMap: removing container" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerName="main"
Oct 30 18:57:43.901633 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.901610    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerName="main"
Oct 30 18:57:43.901725 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.901696    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" containerName="istio-proxy"
Oct 30 18:57:43.901725 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.901708    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" containerName="main"
Oct 30 18:57:43.906485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.906466    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"
Oct 30 18:57:43.913250 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.913222    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"]
Oct 30 18:57:43.913799 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.913780    4372 cpu_manager.go:414] "containerMap: removing container" podUID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" containerName="istio-proxy"
Oct 30 18:57:43.913799 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.913797    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" containerName="istio-proxy"
Oct 30 18:57:43.913938 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.913902    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" containerName="istio-proxy"
Oct 30 18:57:43.917235 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.917215    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"
Oct 30 18:57:43.918967 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.918947    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-fc28ee\""
Oct 30 18:57:43.919098 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.919079    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-fc28ee-dockercfg-qzt4x\""
Oct 30 18:57:43.921266 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:43.921244    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"]
Oct 30 18:57:44.173150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.173049    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"]
Oct 30 18:57:44.253963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.253929    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8cedc2-6b43-4606-b240-6afa12806893-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb\" (UID: \"4b8cedc2-6b43-4606-b240-6afa12806893\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"
Oct 30 18:57:44.254156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.254116    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82358f53-9e9e-4e7c-a4f9-6dc52c326c35-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w\" (UID: \"82358f53-9e9e-4e7c-a4f9-6dc52c326c35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"
Oct 30 18:57:44.254256 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.254236    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk9kd\" (UniqueName: \"kubernetes.io/projected/4b8cedc2-6b43-4606-b240-6afa12806893-kube-api-access-sk9kd\") pod \"payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb\" (UID: \"4b8cedc2-6b43-4606-b240-6afa12806893\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"
Oct 30 18:57:44.254309 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.254292    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4gjb\" (UniqueName: \"kubernetes.io/projected/82358f53-9e9e-4e7c-a4f9-6dc52c326c35-kube-api-access-c4gjb\") pod \"payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w\" (UID: \"82358f53-9e9e-4e7c-a4f9-6dc52c326c35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"
Oct 30 18:57:44.268177 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.268118    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae2b494-2225-49bb-a1a5-2e45bc8cc12d" path="/var/lib/kubelet/pods/0ae2b494-2225-49bb-a1a5-2e45bc8cc12d/volumes"
Oct 30 18:57:44.268780 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.268763    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57b527cf-ac31-44b7-8ff0-6e40ee60ce32" path="/var/lib/kubelet/pods/57b527cf-ac31-44b7-8ff0-6e40ee60ce32/volumes"
Oct 30 18:57:44.355790 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.355760    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8cedc2-6b43-4606-b240-6afa12806893-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb\" (UID: \"4b8cedc2-6b43-4606-b240-6afa12806893\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"
Oct 30 18:57:44.355941 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.355865    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82358f53-9e9e-4e7c-a4f9-6dc52c326c35-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w\" (UID: \"82358f53-9e9e-4e7c-a4f9-6dc52c326c35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"
Oct 30 18:57:44.355941 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.355924    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk9kd\" (UniqueName: \"kubernetes.io/projected/4b8cedc2-6b43-4606-b240-6afa12806893-kube-api-access-sk9kd\") pod \"payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb\" (UID: \"4b8cedc2-6b43-4606-b240-6afa12806893\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"
Oct 30 18:57:44.356066 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.355973    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4gjb\" (UniqueName: \"kubernetes.io/projected/82358f53-9e9e-4e7c-a4f9-6dc52c326c35-kube-api-access-c4gjb\") pod \"payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w\" (UID: \"82358f53-9e9e-4e7c-a4f9-6dc52c326c35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"
Oct 30 18:57:44.356617 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.356592    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8cedc2-6b43-4606-b240-6afa12806893-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb\" (UID: \"4b8cedc2-6b43-4606-b240-6afa12806893\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"
Oct 30 18:57:44.356617 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.356603    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82358f53-9e9e-4e7c-a4f9-6dc52c326c35-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w\" (UID: \"82358f53-9e9e-4e7c-a4f9-6dc52c326c35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"
Oct 30 18:57:44.372294 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.372268    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk9kd\" (UniqueName: \"kubernetes.io/projected/4b8cedc2-6b43-4606-b240-6afa12806893-kube-api-access-sk9kd\") pod \"payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb\" (UID: \"4b8cedc2-6b43-4606-b240-6afa12806893\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"
Oct 30 18:57:44.372389 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.372340    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4gjb\" (UniqueName: \"kubernetes.io/projected/82358f53-9e9e-4e7c-a4f9-6dc52c326c35-kube-api-access-c4gjb\") pod \"payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w\" (UID: \"82358f53-9e9e-4e7c-a4f9-6dc52c326c35\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"
Oct 30 18:57:44.521627 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.521533    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"
Oct 30 18:57:44.665522 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.665493    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"
Oct 30 18:57:44.815825 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.815792    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"]
Oct 30 18:57:44.947649 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:57:44.947625    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8cedc2_6b43_4606_b240_6afa12806893.slice/crio-cb139395a4c73c069bfc4c1d7e9e565d32336ffce050d42ac7c1df1d658effbd WatchSource:0}: Error finding container cb139395a4c73c069bfc4c1d7e9e565d32336ffce050d42ac7c1df1d658effbd: Status 404 returned error can't find the container with id cb139395a4c73c069bfc4c1d7e9e565d32336ffce050d42ac7c1df1d658effbd
Oct 30 18:57:44.948699 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:44.948675    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"]
Oct 30 18:57:45.667273 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:45.667238    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb" event={"ID":"4b8cedc2-6b43-4606-b240-6afa12806893","Type":"ContainerStarted","Data":"5cc386b1bfa81755a0553845f3f8f9be0ced506656582cd6a2a928dd5246f72d"}
Oct 30 18:57:45.667461 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:45.667293    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb" event={"ID":"4b8cedc2-6b43-4606-b240-6afa12806893","Type":"ContainerStarted","Data":"cb139395a4c73c069bfc4c1d7e9e565d32336ffce050d42ac7c1df1d658effbd"}
Oct 30 18:57:45.667461 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:45.667446    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"
Oct 30 18:57:45.669017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:45.668990    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w" event={"ID":"82358f53-9e9e-4e7c-a4f9-6dc52c326c35","Type":"ContainerStarted","Data":"9748a6601ae0ef76f082451ad7c5d25d684a9818839ddfc758cf28580f79e1a5"}
Oct 30 18:57:45.669190 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:45.669034    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w" event={"ID":"82358f53-9e9e-4e7c-a4f9-6dc52c326c35","Type":"ContainerStarted","Data":"a485d7b039344e9aaf5c56e21503ecd729ca6f45387ba853fc0d7c188a0ba115"}
Oct 30 18:57:45.669190 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:45.669085    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"
Oct 30 18:57:45.688271 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:45.688228    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb" podStartSLOduration=2.688216858 podStartE2EDuration="2.688216858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:45.686942112 +0000 UTC m=+1479.555360438" watchObservedRunningTime="2026-10-30 18:57:45.688216858 +0000 UTC m=+1479.556635188"
Oct 30 18:57:45.711594 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:45.711553    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w" podStartSLOduration=2.711541621 podStartE2EDuration="2.711541621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:45.709061527 +0000 UTC m=+1479.577479858" watchObservedRunningTime="2026-10-30 18:57:45.711541621 +0000 UTC m=+1479.579959972"
Oct 30 18:57:50.620168 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:50.620072    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"
Oct 30 18:57:50.620657 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:50.620208    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"
Oct 30 18:57:51.534308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:51.534274    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"]
Oct 30 18:57:51.539739 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:51.539714    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:51.553098 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:51.553072    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-fc28ee-serving-cert\""
Oct 30 18:57:51.557871 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:51.557841    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"]
Oct 30 18:57:51.579118 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:51.579086    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:51.579408 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:51.579387    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-metrics-certs\") pod \"maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:51.579477 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:51.579452    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea33820-3049-447c-9369-6dd72bd8c85d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:51.579553 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:51.579538    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7f84\" (UniqueName: \"kubernetes.io/projected/0ea33820-3049-447c-9369-6dd72bd8c85d-kube-api-access-v7f84\") pod \"maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:51.688225 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:51.688182    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-metrics-certs\") pod \"maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:51.688763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:51.688282    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea33820-3049-447c-9369-6dd72bd8c85d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:51.688763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:51.688347    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7f84\" (UniqueName: \"kubernetes.io/projected/0ea33820-3049-447c-9369-6dd72bd8c85d-kube-api-access-v7f84\") pod \"maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:51.688763 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:51.688438    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:51.688952 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:57:51.688880    4372 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-serving-cert: secret "maas-api-e2e-praxis-a-fc28ee-serving-cert" not found
Oct 30 18:57:51.689011 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:57:51.688964    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-maas-api-tls podName:0ea33820-3049-447c-9369-6dd72bd8c85d nodeName:}" failed. No retries permitted until 2026-10-30 18:57:52.188940664 +0000 UTC m=+1486.057358975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-maas-api-tls") pod "maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5" (UID: "0ea33820-3049-447c-9369-6dd72bd8c85d") : secret "maas-api-e2e-praxis-a-fc28ee-serving-cert" not found
Oct 30 18:57:51.689229 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:51.689198    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea33820-3049-447c-9369-6dd72bd8c85d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:51.720148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:51.716789    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-metrics-certs\") pod \"maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:51.730148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:51.721063    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7f84\" (UniqueName: \"kubernetes.io/projected/0ea33820-3049-447c-9369-6dd72bd8c85d-kube-api-access-v7f84\") pod \"maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:52.202675 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:52.195441    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:52.226383 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:52.219399    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:52.467835 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:52.467749    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:52.810183 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:52.794103    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"]
Oct 30 18:57:52.824713 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:57:52.824262    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea33820_3049_447c_9369_6dd72bd8c85d.slice/crio-11a5d25bd7a7cf2456b37559d68831ad5fae26677919760ec746e765ad11ba00 WatchSource:0}: Error finding container 11a5d25bd7a7cf2456b37559d68831ad5fae26677919760ec746e765ad11ba00: Status 404 returned error can't find the container with id 11a5d25bd7a7cf2456b37559d68831ad5fae26677919760ec746e765ad11ba00
Oct 30 18:57:53.835195 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:53.835155    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5" event={"ID":"0ea33820-3049-447c-9369-6dd72bd8c85d","Type":"ContainerStarted","Data":"780ceb3f28a06c44f4ba360f466a26a2aa21b3a22985add4642c923ace928a6f"}
Oct 30 18:57:53.835592 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:53.835212    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5" event={"ID":"0ea33820-3049-447c-9369-6dd72bd8c85d","Type":"ContainerStarted","Data":"11a5d25bd7a7cf2456b37559d68831ad5fae26677919760ec746e765ad11ba00"}
Oct 30 18:57:53.835592 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:53.835353    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:57:53.861490 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:53.861451    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5" podStartSLOduration=2.367168584 podStartE2EDuration="2.861439522s" totalImagesPullingTime="494.270938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:53.85964553 +0000 UTC m=+1487.728063872" watchObservedRunningTime="2026-10-30 18:57:53.861439522 +0000 UTC m=+1487.729857894"
Oct 30 18:57:56.678249 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:56.678219    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"
Oct 30 18:57:56.678647 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:56.678286    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"
Oct 30 18:57:57.710864 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:57:57.710821    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:57:59.847799 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:57:59.847770    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:58:09.436790 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.436755    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"]
Oct 30 18:58:09.437385 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.436958    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46" podUID="1dbac43e-9151-4838-928a-f3af21c8f377" containerName="maas-api" containerID="cri-o://26f69ca6a56b865a09541fa3229cf807efa59748b275994cb2662278853359d6" gracePeriod=30
Oct 30 18:58:09.584032 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.583997    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"]
Oct 30 18:58:09.584237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.584220    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg" podUID="1da28a54-d60a-49c7-b8ff-f2ca79f5636f" containerName="payload-processing" containerID="cri-o://f47e3d49c81dfe9d34c14b4fa51e62650bc0aac481137da501f0323cdb6617ff" gracePeriod=30
Oct 30 18:58:09.611489 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.611453    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"]
Oct 30 18:58:09.611928 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.611892    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd" podUID="daef0fb1-64c9-4e07-aaa3-66eb205cc648" containerName="payload-pre-processing" containerID="cri-o://8c5743fd80fdc5ad8f2ca00eb4b3cfe8466a19d7b47c5a3d13ebf869045c460b" gracePeriod=30
Oct 30 18:58:09.807257 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.807227    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:58:09.884909 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.884317    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dbac43e-9151-4838-928a-f3af21c8f377-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dbac43e-9151-4838-928a-f3af21c8f377-openshift-service-ca\") pod \"1dbac43e-9151-4838-928a-f3af21c8f377\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") "
Oct 30 18:58:09.884909 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.884405    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbac43e-9151-4838-928a-f3af21c8f377-kube-api-access-f5lqf\" (UniqueName: \"kubernetes.io/projected/1dbac43e-9151-4838-928a-f3af21c8f377-kube-api-access-f5lqf\") pod \"1dbac43e-9151-4838-928a-f3af21c8f377\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") "
Oct 30 18:58:09.884909 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.884587    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dbac43e-9151-4838-928a-f3af21c8f377-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dbac43e-9151-4838-928a-f3af21c8f377-metrics-certs\") pod \"1dbac43e-9151-4838-928a-f3af21c8f377\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") "
Oct 30 18:58:09.884909 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.884627    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dbac43e-9151-4838-928a-f3af21c8f377-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dbac43e-9151-4838-928a-f3af21c8f377-maas-api-tls\") pod \"1dbac43e-9151-4838-928a-f3af21c8f377\" (UID: \"1dbac43e-9151-4838-928a-f3af21c8f377\") "
Oct 30 18:58:09.910593 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.910366    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbac43e-9151-4838-928a-f3af21c8f377-openshift-service-ca" pod "1dbac43e-9151-4838-928a-f3af21c8f377" (UID: "1dbac43e-9151-4838-928a-f3af21c8f377"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:09.943155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.935607    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dbac43e-9151-4838-928a-f3af21c8f377-maas-api-tls" pod "1dbac43e-9151-4838-928a-f3af21c8f377" (UID: "1dbac43e-9151-4838-928a-f3af21c8f377"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:09.943155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.935739    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbac43e-9151-4838-928a-f3af21c8f377-kube-api-access-f5lqf" pod "1dbac43e-9151-4838-928a-f3af21c8f377" (UID: "1dbac43e-9151-4838-928a-f3af21c8f377"). InnerVolumeSpecName "kube-api-access-f5lqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:09.999008 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.993501    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dbac43e-9151-4838-928a-f3af21c8f377-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:09.999008 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.993536    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dbac43e-9151-4838-928a-f3af21c8f377-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:09.999008 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:09.993549    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5lqf\" (UniqueName: \"kubernetes.io/projected/1dbac43e-9151-4838-928a-f3af21c8f377-kube-api-access-f5lqf\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:10.037174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.035416    4372 generic.go:356] "Generic (PLEG): container finished" podID="1dbac43e-9151-4838-928a-f3af21c8f377" containerID="26f69ca6a56b865a09541fa3229cf807efa59748b275994cb2662278853359d6" exitCode=0
Oct 30 18:58:10.037174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.035643    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46" event={"ID":"1dbac43e-9151-4838-928a-f3af21c8f377","Type":"ContainerDied","Data":"26f69ca6a56b865a09541fa3229cf807efa59748b275994cb2662278853359d6"}
Oct 30 18:58:10.037174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.035697    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46" event={"ID":"1dbac43e-9151-4838-928a-f3af21c8f377","Type":"ContainerDied","Data":"a6038aab7cd9c15985b14116876452bba00cbb27901f066983e177e12396df7f"}
Oct 30 18:58:10.037174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.035729    4372 scope.go:122] "RemoveContainer" containerID="26f69ca6a56b865a09541fa3229cf807efa59748b275994cb2662278853359d6"
Oct 30 18:58:10.037174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.035893    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"
Oct 30 18:58:10.065199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.063909    4372 generic.go:356] "Generic (PLEG): container finished" podID="daef0fb1-64c9-4e07-aaa3-66eb205cc648" containerID="8c5743fd80fdc5ad8f2ca00eb4b3cfe8466a19d7b47c5a3d13ebf869045c460b" exitCode=0
Oct 30 18:58:10.065199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.064107    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd" event={"ID":"daef0fb1-64c9-4e07-aaa3-66eb205cc648","Type":"ContainerDied","Data":"8c5743fd80fdc5ad8f2ca00eb4b3cfe8466a19d7b47c5a3d13ebf869045c460b"}
Oct 30 18:58:10.087185 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.067199    4372 generic.go:356] "Generic (PLEG): container finished" podID="1da28a54-d60a-49c7-b8ff-f2ca79f5636f" containerID="f47e3d49c81dfe9d34c14b4fa51e62650bc0aac481137da501f0323cdb6617ff" exitCode=0
Oct 30 18:58:10.087185 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.067241    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg" event={"ID":"1da28a54-d60a-49c7-b8ff-f2ca79f5636f","Type":"ContainerDied","Data":"f47e3d49c81dfe9d34c14b4fa51e62650bc0aac481137da501f0323cdb6617ff"}
Oct 30 18:58:10.092239 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.091936    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dbac43e-9151-4838-928a-f3af21c8f377-metrics-certs" pod "1dbac43e-9151-4838-928a-f3af21c8f377" (UID: "1dbac43e-9151-4838-928a-f3af21c8f377"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:10.098581 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.096616    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dbac43e-9151-4838-928a-f3af21c8f377-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:10.129458 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.129272    4372 scope.go:122] "RemoveContainer" containerID="26f69ca6a56b865a09541fa3229cf807efa59748b275994cb2662278853359d6"
Oct 30 18:58:10.133069 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:58:10.132980    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f69ca6a56b865a09541fa3229cf807efa59748b275994cb2662278853359d6\": container with ID starting with 26f69ca6a56b865a09541fa3229cf807efa59748b275994cb2662278853359d6 not found: ID does not exist" containerID="26f69ca6a56b865a09541fa3229cf807efa59748b275994cb2662278853359d6"
Oct 30 18:58:10.133069 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.133026    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f69ca6a56b865a09541fa3229cf807efa59748b275994cb2662278853359d6"} err="failed to get container status \"26f69ca6a56b865a09541fa3229cf807efa59748b275994cb2662278853359d6\": rpc error: code = NotFound desc = could not find container \"26f69ca6a56b865a09541fa3229cf807efa59748b275994cb2662278853359d6\": container with ID starting with 26f69ca6a56b865a09541fa3229cf807efa59748b275994cb2662278853359d6 not found: ID does not exist"
Oct 30 18:58:10.151778 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.151749    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"
Oct 30 18:58:10.195170 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.193369    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"
Oct 30 18:58:10.198092 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.197762    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da28a54-d60a-49c7-b8ff-f2ca79f5636f-kube-api-access-9xhpg\" (UniqueName: \"kubernetes.io/projected/1da28a54-d60a-49c7-b8ff-f2ca79f5636f-kube-api-access-9xhpg\") pod \"1da28a54-d60a-49c7-b8ff-f2ca79f5636f\" (UID: \"1da28a54-d60a-49c7-b8ff-f2ca79f5636f\") "
Oct 30 18:58:10.198092 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.197848    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da28a54-d60a-49c7-b8ff-f2ca79f5636f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da28a54-d60a-49c7-b8ff-f2ca79f5636f-plugins-config-volume\") pod \"1da28a54-d60a-49c7-b8ff-f2ca79f5636f\" (UID: \"1da28a54-d60a-49c7-b8ff-f2ca79f5636f\") "
Oct 30 18:58:10.211953 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.199416    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da28a54-d60a-49c7-b8ff-f2ca79f5636f-plugins-config-volume" pod "1da28a54-d60a-49c7-b8ff-f2ca79f5636f" (UID: "1da28a54-d60a-49c7-b8ff-f2ca79f5636f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:10.217342 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.217271    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da28a54-d60a-49c7-b8ff-f2ca79f5636f-kube-api-access-9xhpg" pod "1da28a54-d60a-49c7-b8ff-f2ca79f5636f" (UID: "1da28a54-d60a-49c7-b8ff-f2ca79f5636f"). InnerVolumeSpecName "kube-api-access-9xhpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:10.300371 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.299631    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daef0fb1-64c9-4e07-aaa3-66eb205cc648-kube-api-access-s9pzl\" (UniqueName: \"kubernetes.io/projected/daef0fb1-64c9-4e07-aaa3-66eb205cc648-kube-api-access-s9pzl\") pod \"daef0fb1-64c9-4e07-aaa3-66eb205cc648\" (UID: \"daef0fb1-64c9-4e07-aaa3-66eb205cc648\") "
Oct 30 18:58:10.300371 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.299705    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daef0fb1-64c9-4e07-aaa3-66eb205cc648-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daef0fb1-64c9-4e07-aaa3-66eb205cc648-plugins-config-volume\") pod \"daef0fb1-64c9-4e07-aaa3-66eb205cc648\" (UID: \"daef0fb1-64c9-4e07-aaa3-66eb205cc648\") "
Oct 30 18:58:10.300622 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.300403    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xhpg\" (UniqueName: \"kubernetes.io/projected/1da28a54-d60a-49c7-b8ff-f2ca79f5636f-kube-api-access-9xhpg\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:10.300622 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.300426    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da28a54-d60a-49c7-b8ff-f2ca79f5636f-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:10.312199 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.306544    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daef0fb1-64c9-4e07-aaa3-66eb205cc648-kube-api-access-s9pzl" pod "daef0fb1-64c9-4e07-aaa3-66eb205cc648" (UID: "daef0fb1-64c9-4e07-aaa3-66eb205cc648"). InnerVolumeSpecName "kube-api-access-s9pzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:10.322772 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.319863    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daef0fb1-64c9-4e07-aaa3-66eb205cc648-plugins-config-volume" pod "daef0fb1-64c9-4e07-aaa3-66eb205cc648" (UID: "daef0fb1-64c9-4e07-aaa3-66eb205cc648"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:10.376848 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.376820    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"]
Oct 30 18:58:10.385069 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.385048    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-50b19d-77ffd9ddd6-26p46"]
Oct 30 18:58:10.402269 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.402248    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9pzl\" (UniqueName: \"kubernetes.io/projected/daef0fb1-64c9-4e07-aaa3-66eb205cc648-kube-api-access-s9pzl\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:10.402269 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:10.402269    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daef0fb1-64c9-4e07-aaa3-66eb205cc648-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:11.078798 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:11.078760    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg" event={"ID":"1da28a54-d60a-49c7-b8ff-f2ca79f5636f","Type":"ContainerDied","Data":"2c028c93ea046d6b73f95a2cd497c93e8fbe98d655d7bb047d520229707201dd"}
Oct 30 18:58:11.079681 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:11.079656    4372 scope.go:122] "RemoveContainer" containerID="f47e3d49c81dfe9d34c14b4fa51e62650bc0aac481137da501f0323cdb6617ff"
Oct 30 18:58:11.080169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:11.080148    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"
Oct 30 18:58:11.090839 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:11.090792    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd" event={"ID":"daef0fb1-64c9-4e07-aaa3-66eb205cc648","Type":"ContainerDied","Data":"f9fe1c5177dd60e478ed1a3d9c57461b2da89f2ef599de83829fe432d3411143"}
Oct 30 18:58:11.091159 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:11.091115    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"
Oct 30 18:58:11.129793 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:11.129755    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"]
Oct 30 18:58:11.155958 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:11.155926    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50b19d-6b97c69f9b-d6tdg"]
Oct 30 18:58:11.172163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:11.168304    4372 scope.go:122] "RemoveContainer" containerID="8c5743fd80fdc5ad8f2ca00eb4b3cfe8466a19d7b47c5a3d13ebf869045c460b"
Oct 30 18:58:11.191512 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:11.191479    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:58:11.226239 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:11.226205    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"]
Oct 30 18:58:11.256265 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:11.253671    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-50b19d-664d5c6c9f-wlljd"]
Oct 30 18:58:12.302163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:12.302026    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da28a54-d60a-49c7-b8ff-f2ca79f5636f" path="/var/lib/kubelet/pods/1da28a54-d60a-49c7-b8ff-f2ca79f5636f/volumes"
Oct 30 18:58:12.313150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:12.302773    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dbac43e-9151-4838-928a-f3af21c8f377" path="/var/lib/kubelet/pods/1dbac43e-9151-4838-928a-f3af21c8f377/volumes"
Oct 30 18:58:12.319626 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:12.317733    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daef0fb1-64c9-4e07-aaa3-66eb205cc648" path="/var/lib/kubelet/pods/daef0fb1-64c9-4e07-aaa3-66eb205cc648/volumes"
Oct 30 18:58:14.373481 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:14.319936    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:14.565290 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:14.565229    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:14.956153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:14.937202    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:15.047376 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:15.046252    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:15.107151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:15.101285    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:15.358155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:15.353419    4372 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:58:15.358155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:15.353493    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:15.367435 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:15.367395    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:15.404546 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:15.404490    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 18:58:15.436150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:15.435233    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:58:15.500234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:15.493644    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 18:58:15.576736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:15.569052    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 18:58:15.576736 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:15.569630    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:58:15.743216 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:15.716899    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:58:16.803654 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.803621    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"]
Oct 30 18:58:16.804237 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.804221    4372 cpu_manager.go:414] "containerMap: removing container" podUID="1dbac43e-9151-4838-928a-f3af21c8f377" containerName="maas-api"
Oct 30 18:58:16.804299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.804237    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dbac43e-9151-4838-928a-f3af21c8f377" containerName="maas-api"
Oct 30 18:58:16.804299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.804277    4372 cpu_manager.go:414] "containerMap: removing container" podUID="1da28a54-d60a-49c7-b8ff-f2ca79f5636f" containerName="payload-processing"
Oct 30 18:58:16.804299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.804283    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da28a54-d60a-49c7-b8ff-f2ca79f5636f" containerName="payload-processing"
Oct 30 18:58:16.804299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.804293    4372 cpu_manager.go:414] "containerMap: removing container" podUID="daef0fb1-64c9-4e07-aaa3-66eb205cc648" containerName="payload-pre-processing"
Oct 30 18:58:16.804299 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.804298    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daef0fb1-64c9-4e07-aaa3-66eb205cc648" containerName="payload-pre-processing"
Oct 30 18:58:16.804472 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.804386    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dbac43e-9151-4838-928a-f3af21c8f377" containerName="maas-api"
Oct 30 18:58:16.804472 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.804397    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da28a54-d60a-49c7-b8ff-f2ca79f5636f" containerName="payload-processing"
Oct 30 18:58:16.804472 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.804404    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="daef0fb1-64c9-4e07-aaa3-66eb205cc648" containerName="payload-pre-processing"
Oct 30 18:58:16.809236 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.809217    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"
Oct 30 18:58:16.818389 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.818368    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-bcb5af-dockercfg-s7dqz\""
Oct 30 18:58:16.818485 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.818425    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-bcb5af\""
Oct 30 18:58:16.821515 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.821488    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"]
Oct 30 18:58:16.857618 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.857583    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"]
Oct 30 18:58:16.904092 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.904063    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"
Oct 30 18:58:16.949591 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:16.949556    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"]
Oct 30 18:58:17.011977 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.011945    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfr7q\" (UniqueName: \"kubernetes.io/projected/8d81ca16-a936-4c2b-adee-174cdc1e3266-kube-api-access-rfr7q\") pod \"payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6\" (UID: \"8d81ca16-a936-4c2b-adee-174cdc1e3266\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"
Oct 30 18:58:17.050347 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.041953    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d81ca16-a936-4c2b-adee-174cdc1e3266-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6\" (UID: \"8d81ca16-a936-4c2b-adee-174cdc1e3266\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"
Oct 30 18:58:17.142921 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.142890    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7zzk\" (UniqueName: \"kubernetes.io/projected/6f308cee-8550-48a4-b417-0f49cde47af8-kube-api-access-b7zzk\") pod \"payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z\" (UID: \"6f308cee-8550-48a4-b417-0f49cde47af8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"
Oct 30 18:58:17.143111 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.142977    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfr7q\" (UniqueName: \"kubernetes.io/projected/8d81ca16-a936-4c2b-adee-174cdc1e3266-kube-api-access-rfr7q\") pod \"payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6\" (UID: \"8d81ca16-a936-4c2b-adee-174cdc1e3266\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"
Oct 30 18:58:17.143111 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.143032    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d81ca16-a936-4c2b-adee-174cdc1e3266-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6\" (UID: \"8d81ca16-a936-4c2b-adee-174cdc1e3266\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"
Oct 30 18:58:17.143367 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.143314    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f308cee-8550-48a4-b417-0f49cde47af8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z\" (UID: \"6f308cee-8550-48a4-b417-0f49cde47af8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"
Oct 30 18:58:17.143868 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.143850    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d81ca16-a936-4c2b-adee-174cdc1e3266-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6\" (UID: \"8d81ca16-a936-4c2b-adee-174cdc1e3266\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"
Oct 30 18:58:17.156160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.156090    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfr7q\" (UniqueName: \"kubernetes.io/projected/8d81ca16-a936-4c2b-adee-174cdc1e3266-kube-api-access-rfr7q\") pod \"payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6\" (UID: \"8d81ca16-a936-4c2b-adee-174cdc1e3266\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"
Oct 30 18:58:17.244552 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.244527    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f308cee-8550-48a4-b417-0f49cde47af8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z\" (UID: \"6f308cee-8550-48a4-b417-0f49cde47af8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"
Oct 30 18:58:17.244675 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.244585    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7zzk\" (UniqueName: \"kubernetes.io/projected/6f308cee-8550-48a4-b417-0f49cde47af8-kube-api-access-b7zzk\") pod \"payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z\" (UID: \"6f308cee-8550-48a4-b417-0f49cde47af8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"
Oct 30 18:58:17.245301 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.245283    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f308cee-8550-48a4-b417-0f49cde47af8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z\" (UID: \"6f308cee-8550-48a4-b417-0f49cde47af8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"
Oct 30 18:58:17.259116 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.259092    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7zzk\" (UniqueName: \"kubernetes.io/projected/6f308cee-8550-48a4-b417-0f49cde47af8-kube-api-access-b7zzk\") pod \"payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z\" (UID: \"6f308cee-8550-48a4-b417-0f49cde47af8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"
Oct 30 18:58:17.331118 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.331046    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"
Oct 30 18:58:17.423960 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.423934    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"
Oct 30 18:58:17.613410 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.613380    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"]
Oct 30 18:58:17.614972 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:58:17.614950    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f308cee_8550_48a4_b417_0f49cde47af8.slice/crio-598d8e00464536690194fac17d659347d1c4efbef2f4556ef7e5a59a35957b35 WatchSource:0}: Error finding container 598d8e00464536690194fac17d659347d1c4efbef2f4556ef7e5a59a35957b35: Status 404 returned error can't find the container with id 598d8e00464536690194fac17d659347d1c4efbef2f4556ef7e5a59a35957b35
Oct 30 18:58:17.730012 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:58:17.729976    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d81ca16_a936_4c2b_adee_174cdc1e3266.slice/crio-caa43a494f7fd5407a7f785a399613f36256f7e6f55bd976d6d2268203afedf6 WatchSource:0}: Error finding container caa43a494f7fd5407a7f785a399613f36256f7e6f55bd976d6d2268203afedf6: Status 404 returned error can't find the container with id caa43a494f7fd5407a7f785a399613f36256f7e6f55bd976d6d2268203afedf6
Oct 30 18:58:17.731097 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:17.731064    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"]
Oct 30 18:58:18.627643 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:18.627603    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6" event={"ID":"8d81ca16-a936-4c2b-adee-174cdc1e3266","Type":"ContainerStarted","Data":"4dfe130bb88fd0e0e3f0a6ab1ca596317fcbf3c136b9438474d10a6c83b7c8ee"}
Oct 30 18:58:18.628094 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:18.627652    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6" event={"ID":"8d81ca16-a936-4c2b-adee-174cdc1e3266","Type":"ContainerStarted","Data":"caa43a494f7fd5407a7f785a399613f36256f7e6f55bd976d6d2268203afedf6"}
Oct 30 18:58:18.628094 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:18.627694    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"
Oct 30 18:58:18.629294 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:18.629268    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z" event={"ID":"6f308cee-8550-48a4-b417-0f49cde47af8","Type":"ContainerStarted","Data":"28fbbc65c9af3f5be9e47ecc385a9edee27113566d6f9677ea4a4acbb8b68108"}
Oct 30 18:58:18.629414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:18.629305    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z" event={"ID":"6f308cee-8550-48a4-b417-0f49cde47af8","Type":"ContainerStarted","Data":"598d8e00464536690194fac17d659347d1c4efbef2f4556ef7e5a59a35957b35"}
Oct 30 18:58:18.629414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:18.629332    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"
Oct 30 18:58:18.649971 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:18.649929    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6" podStartSLOduration=2.649918446 podStartE2EDuration="2.649918446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:18.646114934 +0000 UTC m=+1512.514533276" watchObservedRunningTime="2026-10-30 18:58:18.649918446 +0000 UTC m=+1512.518336775"
Oct 30 18:58:18.668869 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:18.668811    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z" podStartSLOduration=2.668799514 podStartE2EDuration="2.668799514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:18.665724226 +0000 UTC m=+1512.534142582" watchObservedRunningTime="2026-10-30 18:58:18.668799514 +0000 UTC m=+1512.537217861"
Oct 30 18:58:19.357102 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:19.356185    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"]
Oct 30 18:58:19.357102 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:19.356429    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5" podUID="c448eb1a-dfec-4669-a5e7-585087730fbc" containerName="istio-proxy" containerID="cri-o://ef37a6325d5e2820bab4b28935f17235178346e364ef6ad808a70e2296dcb057" gracePeriod=30
Oct 30 18:58:24.071492 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.071457    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"]
Oct 30 18:58:24.078937 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.078913    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:24.088185 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.088161    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-bcb5af-serving-cert\""
Oct 30 18:58:24.089335 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.089304    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"]
Oct 30 18:58:24.090703 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.090367    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcxzh\" (UniqueName: \"kubernetes.io/projected/2dd5599e-0c92-4050-9351-6eef1270a0ae-kube-api-access-vcxzh\") pod \"maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:24.090703 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.090437    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd5599e-0c92-4050-9351-6eef1270a0ae-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:24.090703 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.090524    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-metrics-certs\") pod \"maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:24.090703 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.090650    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:24.192187 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.192113    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:24.192354 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:58:24.192239    4372 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-serving-cert: secret "maas-api-e2e-praxis-b-bcb5af-serving-cert" not found
Oct 30 18:58:24.192354 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.192262    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxzh\" (UniqueName: \"kubernetes.io/projected/2dd5599e-0c92-4050-9351-6eef1270a0ae-kube-api-access-vcxzh\") pod \"maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:24.192354 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:58:24.192315    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-maas-api-tls podName:2dd5599e-0c92-4050-9351-6eef1270a0ae nodeName:}" failed. No retries permitted until 2026-10-30 18:58:24.69229535 +0000 UTC m=+1518.560713665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-maas-api-tls") pod "maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc" (UID: "2dd5599e-0c92-4050-9351-6eef1270a0ae") : secret "maas-api-e2e-praxis-b-bcb5af-serving-cert" not found
Oct 30 18:58:24.192517 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.192375    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd5599e-0c92-4050-9351-6eef1270a0ae-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:24.192517 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.192464    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-metrics-certs\") pod \"maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:24.193056 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.193030    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd5599e-0c92-4050-9351-6eef1270a0ae-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:24.195005 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.194981    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-metrics-certs\") pod \"maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:24.207498 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.207471    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcxzh\" (UniqueName: \"kubernetes.io/projected/2dd5599e-0c92-4050-9351-6eef1270a0ae-kube-api-access-vcxzh\") pod \"maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:24.372436 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.372399    4372 patch_prober.go:29] interesting pod/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": read tcp 10.133.0.2:41058->10.133.0.138:15021: read: connection reset by peer" start-of-body=
Oct 30 18:58:24.372619 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.372473    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5" podUID="c448eb1a-dfec-4669-a5e7-585087730fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": read tcp 10.133.0.2:41058->10.133.0.138:15021: read: connection reset by peer"
Oct 30 18:58:24.697517 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.697490    4372 generic.go:356] "Generic (PLEG): container finished" podID="c448eb1a-dfec-4669-a5e7-585087730fbc" containerID="ef37a6325d5e2820bab4b28935f17235178346e364ef6ad808a70e2296dcb057" exitCode=0
Oct 30 18:58:24.697635 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.697574    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5" event={"ID":"c448eb1a-dfec-4669-a5e7-585087730fbc","Type":"ContainerDied","Data":"ef37a6325d5e2820bab4b28935f17235178346e364ef6ad808a70e2296dcb057"}
Oct 30 18:58:24.697680 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.697650    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5" event={"ID":"c448eb1a-dfec-4669-a5e7-585087730fbc","Type":"ContainerDied","Data":"6cd8c0f4d3c901f66392f1f022d15fa3c31cb4828d36ee5510fb21fd19149659"}
Oct 30 18:58:24.697730 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.697677    4372 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cd8c0f4d3c901f66392f1f022d15fa3c31cb4828d36ee5510fb21fd19149659"
Oct 30 18:58:24.697730 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.697695    4372 scope.go:122] "RemoveContainer" containerID="9c76113f3c511d2323e748ba0a7d742c6bfbc6d9e77738b9d827f7c4e27fc626"
Oct 30 18:58:24.703210 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.703192    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:58:24.753153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.739574    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:24.794152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.793840    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:24.845499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.842582    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c448eb1a-dfec-4669-a5e7-585087730fbc-kube-api-access-w2n8d\" (UniqueName: \"kubernetes.io/projected/c448eb1a-dfec-4669-a5e7-585087730fbc-kube-api-access-w2n8d\") pod \"c448eb1a-dfec-4669-a5e7-585087730fbc\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") "
Oct 30 18:58:24.845499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.842691    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-credential-socket\") pod \"c448eb1a-dfec-4669-a5e7-585087730fbc\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") "
Oct 30 18:58:24.845499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.842798    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-workload-socket\") pod \"c448eb1a-dfec-4669-a5e7-585087730fbc\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") "
Oct 30 18:58:24.845499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.842885    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c448eb1a-dfec-4669-a5e7-585087730fbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c448eb1a-dfec-4669-a5e7-585087730fbc-istiod-ca-cert\") pod \"c448eb1a-dfec-4669-a5e7-585087730fbc\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") "
Oct 30 18:58:24.845499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.842920    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-data\") pod \"c448eb1a-dfec-4669-a5e7-585087730fbc\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") "
Oct 30 18:58:24.845499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.842944    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-envoy\") pod \"c448eb1a-dfec-4669-a5e7-585087730fbc\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") "
Oct 30 18:58:24.845499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.842982    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-token\" (UniqueName: \"kubernetes.io/projected/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-token\") pod \"c448eb1a-dfec-4669-a5e7-585087730fbc\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") "
Oct 30 18:58:24.845499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.843034    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-podinfo\") pod \"c448eb1a-dfec-4669-a5e7-585087730fbc\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") "
Oct 30 18:58:24.845499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.843101    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-workload-certs\") pod \"c448eb1a-dfec-4669-a5e7-585087730fbc\" (UID: \"c448eb1a-dfec-4669-a5e7-585087730fbc\") "
Oct 30 18:58:24.845499 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.843931    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-data" pod "c448eb1a-dfec-4669-a5e7-585087730fbc" (UID: "c448eb1a-dfec-4669-a5e7-585087730fbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:24.846231 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.846196    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-workload-certs" pod "c448eb1a-dfec-4669-a5e7-585087730fbc" (UID: "c448eb1a-dfec-4669-a5e7-585087730fbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:24.847269 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.846929    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-credential-socket" pod "c448eb1a-dfec-4669-a5e7-585087730fbc" (UID: "c448eb1a-dfec-4669-a5e7-585087730fbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:24.847540 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.847500    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-workload-socket" pod "c448eb1a-dfec-4669-a5e7-585087730fbc" (UID: "c448eb1a-dfec-4669-a5e7-585087730fbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:24.847711 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.847610    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c448eb1a-dfec-4669-a5e7-585087730fbc-istiod-ca-cert" pod "c448eb1a-dfec-4669-a5e7-585087730fbc" (UID: "c448eb1a-dfec-4669-a5e7-585087730fbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:24.852312 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.852276    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c448eb1a-dfec-4669-a5e7-585087730fbc-kube-api-access-w2n8d" pod "c448eb1a-dfec-4669-a5e7-585087730fbc" (UID: "c448eb1a-dfec-4669-a5e7-585087730fbc"). InnerVolumeSpecName "kube-api-access-w2n8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:24.854154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.854112    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-envoy" pod "c448eb1a-dfec-4669-a5e7-585087730fbc" (UID: "c448eb1a-dfec-4669-a5e7-585087730fbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:24.854347 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.854255    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-podinfo" pod "c448eb1a-dfec-4669-a5e7-585087730fbc" (UID: "c448eb1a-dfec-4669-a5e7-585087730fbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:58:24.854347 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.854322    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-token" pod "c448eb1a-dfec-4669-a5e7-585087730fbc" (UID: "c448eb1a-dfec-4669-a5e7-585087730fbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:24.948630 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.948150    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2n8d\" (UniqueName: \"kubernetes.io/projected/c448eb1a-dfec-4669-a5e7-585087730fbc-kube-api-access-w2n8d\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:24.948630 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.948192    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:24.948630 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.948208    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:24.948630 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.948222    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c448eb1a-dfec-4669-a5e7-585087730fbc-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:24.948630 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.948235    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:24.948630 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.948249    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:24.948630 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.948262    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:24.948630 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.948276    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c448eb1a-dfec-4669-a5e7-585087730fbc-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:24.948630 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:24.948290    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c448eb1a-dfec-4669-a5e7-585087730fbc-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:25.000919 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.000888    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:25.283673 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.283643    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"]
Oct 30 18:58:25.284586 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:58:25.284563    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd5599e_0c92_4050_9351_6eef1270a0ae.slice/crio-c7afcb8a6a213687ccfcd0703536dbfdec8e979387628b7a0b5566df2b99aa2d WatchSource:0}: Error finding container c7afcb8a6a213687ccfcd0703536dbfdec8e979387628b7a0b5566df2b99aa2d: Status 404 returned error can't find the container with id c7afcb8a6a213687ccfcd0703536dbfdec8e979387628b7a0b5566df2b99aa2d
Oct 30 18:58:25.492637 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.492551    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"]
Oct 30 18:58:25.493278 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.493255    4372 cpu_manager.go:414] "containerMap: removing container" podUID="c448eb1a-dfec-4669-a5e7-585087730fbc" containerName="istio-proxy"
Oct 30 18:58:25.493278 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.493278    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c448eb1a-dfec-4669-a5e7-585087730fbc" containerName="istio-proxy"
Oct 30 18:58:25.493490 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.493479    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="c448eb1a-dfec-4669-a5e7-585087730fbc" containerName="istio-proxy"
Oct 30 18:58:25.497262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.497242    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.515917 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.515897    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-0a962a-openshift-default-dockercfg-msm6z\""
Oct 30 18:58:25.521958 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.521931    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"]
Oct 30 18:58:25.556443 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.556407    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852083d5-0743-44c0-85fd-0551043be8d7-istio-podinfo\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.556643 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.556460    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852083d5-0743-44c0-85fd-0551043be8d7-istio-token\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.556643 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.556532    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-workload-certs\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.556643 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.556577    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-workload-socket\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.556643 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.556627    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf27p\" (UniqueName: \"kubernetes.io/projected/852083d5-0743-44c0-85fd-0551043be8d7-kube-api-access-kf27p\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.556891 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.556663    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852083d5-0743-44c0-85fd-0551043be8d7-istiod-ca-cert\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.556891 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.556691    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-credential-socket\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.556891 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.556813    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-istio-data\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.557053 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.556980    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-istio-envoy\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.670508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.668428    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852083d5-0743-44c0-85fd-0551043be8d7-istio-podinfo\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.670508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.668482    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852083d5-0743-44c0-85fd-0551043be8d7-istio-token\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.670508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.668524    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-workload-certs\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.670508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.668569    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-workload-socket\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.670508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.668599    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf27p\" (UniqueName: \"kubernetes.io/projected/852083d5-0743-44c0-85fd-0551043be8d7-kube-api-access-kf27p\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.670508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.668635    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852083d5-0743-44c0-85fd-0551043be8d7-istiod-ca-cert\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.670508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.668663    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-credential-socket\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.670508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.668713    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-istio-data\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.670508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.668915    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-istio-envoy\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.670508 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.669993    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-workload-socket\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.687158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.672118    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852083d5-0743-44c0-85fd-0551043be8d7-istiod-ca-cert\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.687158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.672417    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-credential-socket\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.687158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.672733    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-istio-data\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.687158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.673260    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-workload-certs\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.695149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.691119    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852083d5-0743-44c0-85fd-0551043be8d7-istio-podinfo\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.740205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.735689    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852083d5-0743-44c0-85fd-0551043be8d7-istio-token\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.817198 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.816658    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf27p\" (UniqueName: \"kubernetes.io/projected/852083d5-0743-44c0-85fd-0551043be8d7-kube-api-access-kf27p\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.898151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.894713    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc" event={"ID":"2dd5599e-0c92-4050-9351-6eef1270a0ae","Type":"ContainerStarted","Data":"c7afcb8a6a213687ccfcd0703536dbfdec8e979387628b7a0b5566df2b99aa2d"}
Oct 30 18:58:25.899849 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.899821    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"
Oct 30 18:58:25.901432 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.901403    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-istio-envoy\") pod \"e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") " pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:25.955345 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.955185    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"]
Oct 30 18:58:25.960804 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:25.960737    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-50b19d-openshift-default-76c855b745-vvdm5"]
Oct 30 18:58:26.113579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:26.113552    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:26.279790 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:26.279761    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c448eb1a-dfec-4669-a5e7-585087730fbc" path="/var/lib/kubelet/pods/c448eb1a-dfec-4669-a5e7-585087730fbc/volumes"
Oct 30 18:58:26.401434 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:58:26.401403    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852083d5_0743_44c0_85fd_0551043be8d7.slice/crio-c114714a383788dcbb5b4522c0b67804990106f9eb711880d5df996b5e3d85f1 WatchSource:0}: Error finding container c114714a383788dcbb5b4522c0b67804990106f9eb711880d5df996b5e3d85f1: Status 404 returned error can't find the container with id c114714a383788dcbb5b4522c0b67804990106f9eb711880d5df996b5e3d85f1
Oct 30 18:58:26.401771 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:26.401757    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"]
Oct 30 18:58:26.404054 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:26.404021    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:58:26.404208 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:26.404102    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:58:26.404208 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:26.404186    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:58:26.914468 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:26.914418    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc" event={"ID":"2dd5599e-0c92-4050-9351-6eef1270a0ae","Type":"ContainerStarted","Data":"9c7a1bcb86d1ebe3f507ab80d006266af775eedfa6f434ec4d233671be28ceb6"}
Oct 30 18:58:26.914698 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:26.914666    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:26.916345 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:26.916324    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv" event={"ID":"852083d5-0743-44c0-85fd-0551043be8d7","Type":"ContainerStarted","Data":"9e2aa3981d9dff77a42096c37cbc58c4a7c277f468b803c38a4822f1baf8ca9e"}
Oct 30 18:58:26.916452 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:26.916362    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv" event={"ID":"852083d5-0743-44c0-85fd-0551043be8d7","Type":"ContainerStarted","Data":"c114714a383788dcbb5b4522c0b67804990106f9eb711880d5df996b5e3d85f1"}
Oct 30 18:58:26.945986 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:26.945944    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc" podStartSLOduration=2.31622521 podStartE2EDuration="2.945932658s" totalImagesPullingTime="629.707448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:26.943478998 +0000 UTC m=+1520.811897354" watchObservedRunningTime="2026-10-30 18:58:26.945932658 +0000 UTC m=+1520.814350988"
Oct 30 18:58:26.970395 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:26.970355    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv" podStartSLOduration=1.970344356 podStartE2EDuration="1.970344356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:26.967704119 +0000 UTC m=+1520.836122449" watchObservedRunningTime="2026-10-30 18:58:26.970344356 +0000 UTC m=+1520.838762686"
Oct 30 18:58:27.114256 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:27.114074    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:27.166156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:27.166039    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:27.696538 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:58:27.696495    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:58:27.923624 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:27.923565    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:27.924430 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:27.924410    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:58:29.639896 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:29.639861    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"
Oct 30 18:58:29.640436 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:29.640382    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"
Oct 30 18:58:30.831483 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:30.831450    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"]
Oct 30 18:58:30.832163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:30.832137    4372 cpu_manager.go:414] "containerMap: removing container" podUID="c448eb1a-dfec-4669-a5e7-585087730fbc" containerName="istio-proxy"
Oct 30 18:58:30.832163 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:30.832157    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c448eb1a-dfec-4669-a5e7-585087730fbc" containerName="istio-proxy"
Oct 30 18:58:30.832347 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:30.832275    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="c448eb1a-dfec-4669-a5e7-585087730fbc" containerName="istio-proxy"
Oct 30 18:58:30.835762 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:30.835745    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"
Oct 30 18:58:30.844044 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:30.844020    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-0a962a\""
Oct 30 18:58:30.844539 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:30.844521    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"]
Oct 30 18:58:30.845639 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:30.845618    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-0a962a-dockercfg-v77p6\""
Oct 30 18:58:30.848544 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:30.848515    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"
Oct 30 18:58:30.851030 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:30.851009    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"]
Oct 30 18:58:30.981189 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:30.957020    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz4zt\" (UniqueName: \"kubernetes.io/projected/6de6da55-2bdc-4cd2-acbe-6892d37ed4c9-kube-api-access-nz4zt\") pod \"payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv\" (UID: \"6de6da55-2bdc-4cd2-acbe-6892d37ed4c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"
Oct 30 18:58:30.981189 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:30.957274    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b3db43-1f22-4a7e-84e6-941644be057b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm\" (UID: \"f4b3db43-1f22-4a7e-84e6-941644be057b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"
Oct 30 18:58:30.981189 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:30.957425    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bn2j\" (UniqueName: \"kubernetes.io/projected/f4b3db43-1f22-4a7e-84e6-941644be057b-kube-api-access-9bn2j\") pod \"payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm\" (UID: \"f4b3db43-1f22-4a7e-84e6-941644be057b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"
Oct 30 18:58:30.981189 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:30.957464    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de6da55-2bdc-4cd2-acbe-6892d37ed4c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv\" (UID: \"6de6da55-2bdc-4cd2-acbe-6892d37ed4c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"
Oct 30 18:58:30.995651 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:30.995617    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"]
Oct 30 18:58:31.120629 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:31.064041    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bn2j\" (UniqueName: \"kubernetes.io/projected/f4b3db43-1f22-4a7e-84e6-941644be057b-kube-api-access-9bn2j\") pod \"payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm\" (UID: \"f4b3db43-1f22-4a7e-84e6-941644be057b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"
Oct 30 18:58:31.120629 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:31.064109    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de6da55-2bdc-4cd2-acbe-6892d37ed4c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv\" (UID: \"6de6da55-2bdc-4cd2-acbe-6892d37ed4c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"
Oct 30 18:58:31.120629 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:31.064172    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz4zt\" (UniqueName: \"kubernetes.io/projected/6de6da55-2bdc-4cd2-acbe-6892d37ed4c9-kube-api-access-nz4zt\") pod \"payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv\" (UID: \"6de6da55-2bdc-4cd2-acbe-6892d37ed4c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"
Oct 30 18:58:31.120629 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:31.064369    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b3db43-1f22-4a7e-84e6-941644be057b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm\" (UID: \"f4b3db43-1f22-4a7e-84e6-941644be057b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"
Oct 30 18:58:31.120629 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:31.065511    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b3db43-1f22-4a7e-84e6-941644be057b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm\" (UID: \"f4b3db43-1f22-4a7e-84e6-941644be057b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"
Oct 30 18:58:31.120629 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:31.112675    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de6da55-2bdc-4cd2-acbe-6892d37ed4c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv\" (UID: \"6de6da55-2bdc-4cd2-acbe-6892d37ed4c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"
Oct 30 18:58:31.144276 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:31.144246    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz4zt\" (UniqueName: \"kubernetes.io/projected/6de6da55-2bdc-4cd2-acbe-6892d37ed4c9-kube-api-access-nz4zt\") pod \"payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv\" (UID: \"6de6da55-2bdc-4cd2-acbe-6892d37ed4c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"
Oct 30 18:58:31.144653 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:31.144626    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bn2j\" (UniqueName: \"kubernetes.io/projected/f4b3db43-1f22-4a7e-84e6-941644be057b-kube-api-access-9bn2j\") pod \"payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm\" (UID: \"f4b3db43-1f22-4a7e-84e6-941644be057b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"
Oct 30 18:58:31.150842 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:31.150813    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"
Oct 30 18:58:31.295976 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:31.295945    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"
Oct 30 18:58:31.436377 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:58:31.436347    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de6da55_2bdc_4cd2_acbe_6892d37ed4c9.slice/crio-9a8ee020a6bd609a5f74ac41f6e26c0a53b27907fa1255afab82f8ac8af5fc64 WatchSource:0}: Error finding container 9a8ee020a6bd609a5f74ac41f6e26c0a53b27907fa1255afab82f8ac8af5fc64: Status 404 returned error can't find the container with id 9a8ee020a6bd609a5f74ac41f6e26c0a53b27907fa1255afab82f8ac8af5fc64
Oct 30 18:58:31.436482 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:31.436417    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"]
Oct 30 18:58:31.581498 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:31.581471    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"]
Oct 30 18:58:31.582441 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:58:31.582413    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b3db43_1f22_4a7e_84e6_941644be057b.slice/crio-eba6bd3cfb456e027df21012d7005d8ba52c817d89c22d6fb70b595b8b8a72a8 WatchSource:0}: Error finding container eba6bd3cfb456e027df21012d7005d8ba52c817d89c22d6fb70b595b8b8a72a8: Status 404 returned error can't find the container with id eba6bd3cfb456e027df21012d7005d8ba52c817d89c22d6fb70b595b8b8a72a8
Oct 30 18:58:32.021985 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:32.021946    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm" event={"ID":"f4b3db43-1f22-4a7e-84e6-941644be057b","Type":"ContainerStarted","Data":"c1d884b993de777e9e372f43cb9552e53c05c1b9ea3e283b8d0ff51ced32131a"}
Oct 30 18:58:32.022463 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:32.022016    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"
Oct 30 18:58:32.022463 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:32.022038    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm" event={"ID":"f4b3db43-1f22-4a7e-84e6-941644be057b","Type":"ContainerStarted","Data":"eba6bd3cfb456e027df21012d7005d8ba52c817d89c22d6fb70b595b8b8a72a8"}
Oct 30 18:58:32.023596 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:32.023572    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv" event={"ID":"6de6da55-2bdc-4cd2-acbe-6892d37ed4c9","Type":"ContainerStarted","Data":"14d220cf348d93dc6619313241946cba9d7019d4e86e8f21bf5e811ca2e00801"}
Oct 30 18:58:32.023724 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:32.023617    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv" event={"ID":"6de6da55-2bdc-4cd2-acbe-6892d37ed4c9","Type":"ContainerStarted","Data":"9a8ee020a6bd609a5f74ac41f6e26c0a53b27907fa1255afab82f8ac8af5fc64"}
Oct 30 18:58:32.023724 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:32.023695    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"
Oct 30 18:58:32.049514 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:32.049475    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm" podStartSLOduration=2.049463729 podStartE2EDuration="2.049463729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:32.046257012 +0000 UTC m=+1525.914675345" watchObservedRunningTime="2026-10-30 18:58:32.049463729 +0000 UTC m=+1525.917882059"
Oct 30 18:58:32.069773 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:32.069618    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv" podStartSLOduration=2.069610578 podStartE2EDuration="2.069610578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:32.068753605 +0000 UTC m=+1525.937171935" watchObservedRunningTime="2026-10-30 18:58:32.069610578 +0000 UTC m=+1525.938028909"
Oct 30 18:58:32.928727 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:32.928697    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:58:38.302160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.297670    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"]
Oct 30 18:58:38.358373 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.358345    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"]
Oct 30 18:58:38.358561 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.358460    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:38.368971 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.368939    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg7bq\" (UniqueName: \"kubernetes.io/projected/b4875864-541b-423c-9386-3535205fe52a-kube-api-access-rg7bq\") pod \"maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:38.369631 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.369603    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4875864-541b-423c-9386-3535205fe52a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:38.369731 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.369707    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4875864-541b-423c-9386-3535205fe52a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:38.369855 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.369838    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4875864-541b-423c-9386-3535205fe52a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:38.381473 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.381437    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-0a962a-serving-cert\""
Oct 30 18:58:38.470970 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.470944    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4875864-541b-423c-9386-3535205fe52a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:38.471077 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.471033    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4875864-541b-423c-9386-3535205fe52a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:38.471169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.471144    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4875864-541b-423c-9386-3535205fe52a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:38.471231 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.471195    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg7bq\" (UniqueName: \"kubernetes.io/projected/b4875864-541b-423c-9386-3535205fe52a-kube-api-access-rg7bq\") pod \"maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:38.471706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.471681    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4875864-541b-423c-9386-3535205fe52a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:38.473655 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.473636    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4875864-541b-423c-9386-3535205fe52a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:38.473767 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.473669    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4875864-541b-423c-9386-3535205fe52a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:38.484610 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.484590    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg7bq\" (UniqueName: \"kubernetes.io/projected/b4875864-541b-423c-9386-3535205fe52a-kube-api-access-rg7bq\") pod \"maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:38.677153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:38.676441    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:39.285154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:39.281151    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"]
Oct 30 18:58:39.285154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:39.281397    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm" podUID="77adbd2d-552e-4a78-a8b7-1ccf8db4a679" containerName="maas-api" containerID="cri-o://9d8f54b29a537cab8ca12da995dbafc91ef6d4d848dd1b0d312b8607229f5701" gracePeriod=30
Oct 30 18:58:39.868151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:39.847267    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"]
Oct 30 18:58:40.009981 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:39.987738    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:40.081065 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:40.080253    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:40.095172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:40.088335    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:40.186450 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:40.160665    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"]
Oct 30 18:58:40.186450 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:40.160928    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc" podUID="20da1af7-a9e4-45e0-b78e-712910be1c68" containerName="payload-processing" containerID="cri-o://a41cebc3e6de9412bd10e7e8e6fb995ad3c9ae2fc635917a8709d05e4e471db3" gracePeriod=30
Oct 30 18:58:40.337990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:40.337948    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:40.508236 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:40.508180    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:40.613458 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:40.523492    4372 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:58:40.613752 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:40.613699    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" podUID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:40.615184 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:40.593701    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n" podUID="90cac65e-3806-45fe-b407-c953c9a95560" containerName="maas-api" containerID="cri-o://21e8d36c78303f5ea8c2dc9ddcf870caadd66a47db57aba83bf9a449fcb2d736" gracePeriod=30
Oct 30 18:58:40.850875 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:40.819436    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 18:58:40.864978 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:40.853164    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"]
Oct 30 18:58:40.864978 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:40.853248    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"]
Oct 30 18:58:40.864978 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:40.853439    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v" podUID="8db96541-6147-4197-8623-767675ace383" containerName="payload-pre-processing" containerID="cri-o://602e4d3f0ab202eb8ed5cb438232b6edd27cc4dbacf4b46e5dab4937c990c647" gracePeriod=30
Oct 30 18:58:40.996660 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:40.996616    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 18:58:41.018623 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.018597    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:58:41.214554 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.214499    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:41.215455 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.215436    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:58:41.242180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.225408    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:41.242180 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.229409    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:58:41.402273 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.402228    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm" podUID="77adbd2d-552e-4a78-a8b7-1ccf8db4a679" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.134:8443/health\": dial tcp 10.133.0.134:8443: connect: connection refused"
Oct 30 18:58:41.460719 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:58:41.426189    4372 container.go:588] Failed to update stats for container "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20da1af7_a9e4_45e0_b78e_712910be1c68.slice/crio-959cb65b50d80d00557fe44d1d71ee08cb64b52893138d6e6871c92391dbcf82": error while statting cgroup v2: [unable to parse /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20da1af7_a9e4_45e0_b78e_712910be1c68.slice/crio-959cb65b50d80d00557fe44d1d71ee08cb64b52893138d6e6871c92391dbcf82/memory.stat: read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20da1af7_a9e4_45e0_b78e_712910be1c68.slice/crio-959cb65b50d80d00557fe44d1d71ee08cb64b52893138d6e6871c92391dbcf82/memory.stat: no such device], continuing to push stats
Oct 30 18:58:41.494160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.479332    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:58:41.494160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.479912    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:58:41.572686 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.571186    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 18:58:41.823280 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.821890    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 18:58:41.823280 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.822340    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:58:41.897640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.887628    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 18:58:41.897640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.887684    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 18:58:41.897640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.888177    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 18:58:41.908147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.904267    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 18:58:41.908147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.906222    4372 generic.go:356] "Generic (PLEG): container finished" podID="20da1af7-a9e4-45e0-b78e-712910be1c68" containerID="a41cebc3e6de9412bd10e7e8e6fb995ad3c9ae2fc635917a8709d05e4e471db3" exitCode=0
Oct 30 18:58:41.908147 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.906254    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc" event={"ID":"20da1af7-a9e4-45e0-b78e-712910be1c68","Type":"ContainerDied","Data":"a41cebc3e6de9412bd10e7e8e6fb995ad3c9ae2fc635917a8709d05e4e471db3"}
Oct 30 18:58:41.931960 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.931930    4372 generic.go:356] "Generic (PLEG): container finished" podID="90cac65e-3806-45fe-b407-c953c9a95560" containerID="21e8d36c78303f5ea8c2dc9ddcf870caadd66a47db57aba83bf9a449fcb2d736" exitCode=0
Oct 30 18:58:41.932414 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.932299    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n" event={"ID":"90cac65e-3806-45fe-b407-c953c9a95560","Type":"ContainerDied","Data":"21e8d36c78303f5ea8c2dc9ddcf870caadd66a47db57aba83bf9a449fcb2d736"}
Oct 30 18:58:41.934011 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:58:41.933857    4372 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db96541_6147_4197_8623_767675ace383.slice/crio-602e4d3f0ab202eb8ed5cb438232b6edd27cc4dbacf4b46e5dab4937c990c647.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20da1af7_a9e4_45e0_b78e_712910be1c68.slice/crio-a41cebc3e6de9412bd10e7e8e6fb995ad3c9ae2fc635917a8709d05e4e471db3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20da1af7_a9e4_45e0_b78e_712910be1c68.slice/crio-conmon-a41cebc3e6de9412bd10e7e8e6fb995ad3c9ae2fc635917a8709d05e4e471db3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cac65e_3806_45fe_b407_c953c9a95560.slice/crio-21e8d36c78303f5ea8c2dc9ddcf870caadd66a47db57aba83bf9a449fcb2d736.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db96541_6147_4197_8623_767675ace383.slice/crio-conmon-602e4d3f0ab202eb8ed5cb438232b6edd27cc4dbacf4b46e5dab4937c990c647.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cac65e_3806_45fe_b407_c953c9a95560.slice/crio-conmon-21e8d36c78303f5ea8c2dc9ddcf870caadd66a47db57aba83bf9a449fcb2d736.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 18:58:41.937646 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.937627    4372 generic.go:356] "Generic (PLEG): container finished" podID="8db96541-6147-4197-8623-767675ace383" containerID="602e4d3f0ab202eb8ed5cb438232b6edd27cc4dbacf4b46e5dab4937c990c647" exitCode=0
Oct 30 18:58:41.937908 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.937887    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v" event={"ID":"8db96541-6147-4197-8623-767675ace383","Type":"ContainerDied","Data":"602e4d3f0ab202eb8ed5cb438232b6edd27cc4dbacf4b46e5dab4937c990c647"}
Oct 30 18:58:41.947864 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.947838    4372 generic.go:356] "Generic (PLEG): container finished" podID="77adbd2d-552e-4a78-a8b7-1ccf8db4a679" containerID="9d8f54b29a537cab8ca12da995dbafc91ef6d4d848dd1b0d312b8607229f5701" exitCode=0
Oct 30 18:58:41.948081 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.948062    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm" event={"ID":"77adbd2d-552e-4a78-a8b7-1ccf8db4a679","Type":"ContainerDied","Data":"9d8f54b29a537cab8ca12da995dbafc91ef6d4d848dd1b0d312b8607229f5701"}
Oct 30 18:58:41.950573 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:41.950544    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l" event={"ID":"b4875864-541b-423c-9386-3535205fe52a","Type":"ContainerStarted","Data":"957c29d114bfbf9be9af52444f6f1abc571e685cd5c2e5ef3277bb725370beb6"}
Oct 30 18:58:42.040552 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.040244    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"
Oct 30 18:58:42.102696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.102656    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:58:42.125297 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.125271    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-metrics-certs\") pod \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") "
Oct 30 18:58:42.125429 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.125394    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8db96541-6147-4197-8623-767675ace383-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db96541-6147-4197-8623-767675ace383-plugins-config-volume\") pod \"8db96541-6147-4197-8623-767675ace383\" (UID: \"8db96541-6147-4197-8623-767675ace383\") "
Oct 30 18:58:42.125500 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.125448    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-openshift-service-ca\") pod \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") "
Oct 30 18:58:42.125552 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.125535    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8db96541-6147-4197-8623-767675ace383-kube-api-access-2jlgg\" (UniqueName: \"kubernetes.io/projected/8db96541-6147-4197-8623-767675ace383-kube-api-access-2jlgg\") pod \"8db96541-6147-4197-8623-767675ace383\" (UID: \"8db96541-6147-4197-8623-767675ace383\") "
Oct 30 18:58:42.125671 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.125622    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-kube-api-access-cczc6\" (UniqueName: \"kubernetes.io/projected/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-kube-api-access-cczc6\") pod \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") "
Oct 30 18:58:42.125755 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.125712    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-maas-api-tls\") pod \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\" (UID: \"77adbd2d-552e-4a78-a8b7-1ccf8db4a679\") "
Oct 30 18:58:42.125878 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.125852    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-openshift-service-ca" pod "77adbd2d-552e-4a78-a8b7-1ccf8db4a679" (UID: "77adbd2d-552e-4a78-a8b7-1ccf8db4a679"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:42.126367 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.126345    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db96541-6147-4197-8623-767675ace383-plugins-config-volume" pod "8db96541-6147-4197-8623-767675ace383" (UID: "8db96541-6147-4197-8623-767675ace383"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:42.126517 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.126454    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:42.128323 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.128297    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-metrics-certs" pod "77adbd2d-552e-4a78-a8b7-1ccf8db4a679" (UID: "77adbd2d-552e-4a78-a8b7-1ccf8db4a679"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:42.128422 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.127776    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"
Oct 30 18:58:42.128627 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.128608    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db96541-6147-4197-8623-767675ace383-kube-api-access-2jlgg" pod "8db96541-6147-4197-8623-767675ace383" (UID: "8db96541-6147-4197-8623-767675ace383"). InnerVolumeSpecName "kube-api-access-2jlgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:42.128706 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.128631    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-kube-api-access-cczc6" pod "77adbd2d-552e-4a78-a8b7-1ccf8db4a679" (UID: "77adbd2d-552e-4a78-a8b7-1ccf8db4a679"). InnerVolumeSpecName "kube-api-access-cczc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:42.130383 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.130361    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-maas-api-tls" pod "77adbd2d-552e-4a78-a8b7-1ccf8db4a679" (UID: "77adbd2d-552e-4a78-a8b7-1ccf8db4a679"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:42.145562 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.140429    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:58:42.228057 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.228025    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-metrics-certs\") pod \"90cac65e-3806-45fe-b407-c953c9a95560\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") "
Oct 30 18:58:42.228259 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.228098    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90cac65e-3806-45fe-b407-c953c9a95560-kube-api-access-vbvhd\" (UniqueName: \"kubernetes.io/projected/90cac65e-3806-45fe-b407-c953c9a95560-kube-api-access-vbvhd\") pod \"90cac65e-3806-45fe-b407-c953c9a95560\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") "
Oct 30 18:58:42.228259 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.228204    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90cac65e-3806-45fe-b407-c953c9a95560-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90cac65e-3806-45fe-b407-c953c9a95560-openshift-service-ca\") pod \"90cac65e-3806-45fe-b407-c953c9a95560\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") "
Oct 30 18:58:42.228399 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.228375    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-maas-api-tls\") pod \"90cac65e-3806-45fe-b407-c953c9a95560\" (UID: \"90cac65e-3806-45fe-b407-c953c9a95560\") "
Oct 30 18:58:42.228458 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.228408    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20da1af7-a9e4-45e0-b78e-712910be1c68-kube-api-access-2g7kt\" (UniqueName: \"kubernetes.io/projected/20da1af7-a9e4-45e0-b78e-712910be1c68-kube-api-access-2g7kt\") pod \"20da1af7-a9e4-45e0-b78e-712910be1c68\" (UID: \"20da1af7-a9e4-45e0-b78e-712910be1c68\") "
Oct 30 18:58:42.228518 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.228465    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20da1af7-a9e4-45e0-b78e-712910be1c68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20da1af7-a9e4-45e0-b78e-712910be1c68-plugins-config-volume\") pod \"20da1af7-a9e4-45e0-b78e-712910be1c68\" (UID: \"20da1af7-a9e4-45e0-b78e-712910be1c68\") "
Oct 30 18:58:42.228673 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.228651    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90cac65e-3806-45fe-b407-c953c9a95560-openshift-service-ca" pod "90cac65e-3806-45fe-b407-c953c9a95560" (UID: "90cac65e-3806-45fe-b407-c953c9a95560"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:42.229431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.229313    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:42.229431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.229340    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90cac65e-3806-45fe-b407-c953c9a95560-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:42.229431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.229357    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db96541-6147-4197-8623-767675ace383-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:42.229431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.229371    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jlgg\" (UniqueName: \"kubernetes.io/projected/8db96541-6147-4197-8623-767675ace383-kube-api-access-2jlgg\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:42.229431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.229389    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cczc6\" (UniqueName: \"kubernetes.io/projected/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-kube-api-access-cczc6\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:42.229431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.229407    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77adbd2d-552e-4a78-a8b7-1ccf8db4a679-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:42.229431 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.229422    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20da1af7-a9e4-45e0-b78e-712910be1c68-plugins-config-volume" pod "20da1af7-a9e4-45e0-b78e-712910be1c68" (UID: "20da1af7-a9e4-45e0-b78e-712910be1c68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:42.230640 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.230612    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cac65e-3806-45fe-b407-c953c9a95560-kube-api-access-vbvhd" pod "90cac65e-3806-45fe-b407-c953c9a95560" (UID: "90cac65e-3806-45fe-b407-c953c9a95560"). InnerVolumeSpecName "kube-api-access-vbvhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:42.231074 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.231051    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-metrics-certs" pod "90cac65e-3806-45fe-b407-c953c9a95560" (UID: "90cac65e-3806-45fe-b407-c953c9a95560"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:42.231189 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.231071    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-maas-api-tls" pod "90cac65e-3806-45fe-b407-c953c9a95560" (UID: "90cac65e-3806-45fe-b407-c953c9a95560"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:42.231300 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.231282    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20da1af7-a9e4-45e0-b78e-712910be1c68-kube-api-access-2g7kt" pod "20da1af7-a9e4-45e0-b78e-712910be1c68" (UID: "20da1af7-a9e4-45e0-b78e-712910be1c68"). InnerVolumeSpecName "kube-api-access-2g7kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:42.330963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.330925    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:42.330963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.330960    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g7kt\" (UniqueName: \"kubernetes.io/projected/20da1af7-a9e4-45e0-b78e-712910be1c68-kube-api-access-2g7kt\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:42.331250 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.330978    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20da1af7-a9e4-45e0-b78e-712910be1c68-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:42.331250 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.330992    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cac65e-3806-45fe-b407-c953c9a95560-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:42.331250 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.331010    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbvhd\" (UniqueName: \"kubernetes.io/projected/90cac65e-3806-45fe-b407-c953c9a95560-kube-api-access-vbvhd\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:42.965773 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.965740    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm" event={"ID":"77adbd2d-552e-4a78-a8b7-1ccf8db4a679","Type":"ContainerDied","Data":"e579d08e84032c283ca816c2134d0b909c660d34bc96e77f4f71887347cfcc00"}
Oct 30 18:58:42.965946 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.965810    4372 scope.go:122] "RemoveContainer" containerID="9d8f54b29a537cab8ca12da995dbafc91ef6d4d848dd1b0d312b8607229f5701"
Oct 30 18:58:42.966019 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.965955    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"
Oct 30 18:58:42.969203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.969171    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l" event={"ID":"b4875864-541b-423c-9386-3535205fe52a","Type":"ContainerStarted","Data":"04123e157577e9704e44bfb6f63b69b5f31796ce3c662d0120ff08969dc69b53"}
Oct 30 18:58:42.969341 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.969249    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:42.970967 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.970942    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"
Oct 30 18:58:42.971073 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.970977    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc" event={"ID":"20da1af7-a9e4-45e0-b78e-712910be1c68","Type":"ContainerDied","Data":"959cb65b50d80d00557fe44d1d71ee08cb64b52893138d6e6871c92391dbcf82"}
Oct 30 18:58:42.973969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.973107    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"
Oct 30 18:58:42.973969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.973106    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n" event={"ID":"90cac65e-3806-45fe-b407-c953c9a95560","Type":"ContainerDied","Data":"258807eada44cd6e5adc39a4b6eb636516d514d16c404a13bb1dbc4967fa7162"}
Oct 30 18:58:42.977717 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.977695    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"
Oct 30 18:58:42.977840 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.977701    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v" event={"ID":"8db96541-6147-4197-8623-767675ace383","Type":"ContainerDied","Data":"6a98876dbd13ff8199e2467c07265030b8d9d9ea07ede1602488135ecf0eda28"}
Oct 30 18:58:42.982191 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.982177    4372 scope.go:122] "RemoveContainer" containerID="a41cebc3e6de9412bd10e7e8e6fb995ad3c9ae2fc635917a8709d05e4e471db3"
Oct 30 18:58:42.994635 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:42.994612    4372 scope.go:122] "RemoveContainer" containerID="21e8d36c78303f5ea8c2dc9ddcf870caadd66a47db57aba83bf9a449fcb2d736"
Oct 30 18:58:43.002887 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:43.002837    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l" podStartSLOduration=4.075665922 podStartE2EDuration="5.002825592s" totalImagesPullingTime="927.15967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:42.99974508 +0000 UTC m=+1536.868163411" watchObservedRunningTime="2026-10-30 18:58:43.002825592 +0000 UTC m=+1536.871243922"
Oct 30 18:58:43.005900 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:43.005881    4372 scope.go:122] "RemoveContainer" containerID="602e4d3f0ab202eb8ed5cb438232b6edd27cc4dbacf4b46e5dab4937c990c647"
Oct 30 18:58:43.032941 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:43.032917    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"]
Oct 30 18:58:43.034783 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:43.034763    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"
Oct 30 18:58:43.035120 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:43.035095    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"
Oct 30 18:58:43.036785 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:43.036758    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb19ff-588c6f588b-c2xcm"]
Oct 30 18:58:43.049804 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:43.049769    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"]
Oct 30 18:58:43.060034 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:43.060012    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-493721-65765f864d-lgb9n"]
Oct 30 18:58:43.079262 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:43.079239    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"]
Oct 30 18:58:43.086537 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:43.086518    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-493721-7d977b5cf-fx26v"]
Oct 30 18:58:43.102557 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:43.102526    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"]
Oct 30 18:58:43.104522 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:43.104505    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-493721-6dbb8d8996-d57bc"]
Oct 30 18:58:44.271024 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.270987    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20da1af7-a9e4-45e0-b78e-712910be1c68" path="/var/lib/kubelet/pods/20da1af7-a9e4-45e0-b78e-712910be1c68/volumes"
Oct 30 18:58:44.271625 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.271605    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77adbd2d-552e-4a78-a8b7-1ccf8db4a679" path="/var/lib/kubelet/pods/77adbd2d-552e-4a78-a8b7-1ccf8db4a679/volumes"
Oct 30 18:58:44.272022 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.272003    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8db96541-6147-4197-8623-767675ace383" path="/var/lib/kubelet/pods/8db96541-6147-4197-8623-767675ace383/volumes"
Oct 30 18:58:44.272769 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.272748    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90cac65e-3806-45fe-b407-c953c9a95560" path="/var/lib/kubelet/pods/90cac65e-3806-45fe-b407-c953c9a95560/volumes"
Oct 30 18:58:44.640378 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.640350    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"]
Oct 30 18:58:44.640990 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.640969    4372 cpu_manager.go:414] "containerMap: removing container" podUID="8db96541-6147-4197-8623-767675ace383" containerName="payload-pre-processing"
Oct 30 18:58:44.641096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.640992    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8db96541-6147-4197-8623-767675ace383" containerName="payload-pre-processing"
Oct 30 18:58:44.641096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.641007    4372 cpu_manager.go:414] "containerMap: removing container" podUID="90cac65e-3806-45fe-b407-c953c9a95560" containerName="maas-api"
Oct 30 18:58:44.641096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.641014    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90cac65e-3806-45fe-b407-c953c9a95560" containerName="maas-api"
Oct 30 18:58:44.641096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.641041    4372 cpu_manager.go:414] "containerMap: removing container" podUID="20da1af7-a9e4-45e0-b78e-712910be1c68" containerName="payload-processing"
Oct 30 18:58:44.641096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.641048    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20da1af7-a9e4-45e0-b78e-712910be1c68" containerName="payload-processing"
Oct 30 18:58:44.641096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.641078    4372 cpu_manager.go:414] "containerMap: removing container" podUID="77adbd2d-552e-4a78-a8b7-1ccf8db4a679" containerName="maas-api"
Oct 30 18:58:44.641096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.641086    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77adbd2d-552e-4a78-a8b7-1ccf8db4a679" containerName="maas-api"
Oct 30 18:58:44.641472 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.641268    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="8db96541-6147-4197-8623-767675ace383" containerName="payload-pre-processing"
Oct 30 18:58:44.641472 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.641285    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="90cac65e-3806-45fe-b407-c953c9a95560" containerName="maas-api"
Oct 30 18:58:44.641472 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.641297    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="20da1af7-a9e4-45e0-b78e-712910be1c68" containerName="payload-processing"
Oct 30 18:58:44.641472 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.641313    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="77adbd2d-552e-4a78-a8b7-1ccf8db4a679" containerName="maas-api"
Oct 30 18:58:44.646330 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.646311    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.657164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.657147    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-86c150-openshift-default-dockercfg-htf9s\""
Oct 30 18:58:44.665933 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.665912    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"]
Oct 30 18:58:44.789778 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.789744    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-workload-certs\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.789945 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.789852    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-token\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.789945 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.789878    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3028a9d2-a829-4420-a287-ce7c2ea64334-istiod-ca-cert\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.789945 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.789932    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkplh\" (UniqueName: \"kubernetes.io/projected/3028a9d2-a829-4420-a287-ce7c2ea64334-kube-api-access-lkplh\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.790080 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.789963    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-workload-socket\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.790080 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.790007    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-data\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.790190 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.790086    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-podinfo\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.790190 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.790120    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-envoy\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.790301 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.790194    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-credential-socket\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.895224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.893854    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-workload-certs\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.895224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.893985    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-token\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.895224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.894019    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3028a9d2-a829-4420-a287-ce7c2ea64334-istiod-ca-cert\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.895224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.894073    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkplh\" (UniqueName: \"kubernetes.io/projected/3028a9d2-a829-4420-a287-ce7c2ea64334-kube-api-access-lkplh\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.895224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.894105    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-workload-socket\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.895224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.894197    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-data\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.895224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.894237    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-podinfo\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.895224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.894308    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-envoy\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.895224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.894343    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-credential-socket\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.895224 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.894878    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-credential-socket\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.921190 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.908236    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-workload-certs\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.921190 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.909373    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3028a9d2-a829-4420-a287-ce7c2ea64334-istiod-ca-cert\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.921190 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.909890    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-workload-socket\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.921190 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.910212    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-data\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.971297 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.970395    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-envoy\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.971477 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.971383    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkplh\" (UniqueName: \"kubernetes.io/projected/3028a9d2-a829-4420-a287-ce7c2ea64334-kube-api-access-lkplh\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.976914 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.971902    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-token\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:44.976914 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:44.975784    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-podinfo\") pod \"e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") " pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:45.259575 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:45.259508    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:45.556095 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:58:45.556067    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3028a9d2_a829_4420_a287_ce7c2ea64334.slice/crio-163989e600716e8938d221aba1f7ecf84b4a07ee517ebbc3923416029a2755cb WatchSource:0}: Error finding container 163989e600716e8938d221aba1f7ecf84b4a07ee517ebbc3923416029a2755cb: Status 404 returned error can't find the container with id 163989e600716e8938d221aba1f7ecf84b4a07ee517ebbc3923416029a2755cb
Oct 30 18:58:45.556984 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:45.556958    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"]
Oct 30 18:58:45.558636 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:45.558604    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:58:45.558720 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:45.558669    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:58:45.558720 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:45.558700    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:58:46.107812 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:46.107776    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6" event={"ID":"3028a9d2-a829-4420-a287-ce7c2ea64334","Type":"ContainerStarted","Data":"4935fa0b08a5dfb1063a20173ba3824513c0377ebd70401a42d9a3368f6013a8"}
Oct 30 18:58:46.107963 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:46.107832    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6" event={"ID":"3028a9d2-a829-4420-a287-ce7c2ea64334","Type":"ContainerStarted","Data":"163989e600716e8938d221aba1f7ecf84b4a07ee517ebbc3923416029a2755cb"}
Oct 30 18:58:46.136391 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:46.136343    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6" podStartSLOduration=2.136328417 podStartE2EDuration="2.136328417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:46.132779353 +0000 UTC m=+1540.001197706" watchObservedRunningTime="2026-10-30 18:58:46.136328417 +0000 UTC m=+1540.004746748"
Oct 30 18:58:46.334230 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:46.334180    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:46.334419 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:46.334372    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:47.114879 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:47.114848    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:47.115744 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:47.115726    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:58:49.101153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:49.098436    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 18:58:49.837561 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:49.837530    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"]
Oct 30 18:58:49.842097 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:49.842072    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"
Oct 30 18:58:49.852093 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:49.852073    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-86c150\""
Oct 30 18:58:49.861374 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:49.861351    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"]
Oct 30 18:58:49.901198 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:49.901165    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b5f840-1e90-4f4c-84de-99857ceac633-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk\" (UID: \"40b5f840-1e90-4f4c-84de-99857ceac633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"
Oct 30 18:58:49.901340 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:49.901231    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsvck\" (UniqueName: \"kubernetes.io/projected/40b5f840-1e90-4f4c-84de-99857ceac633-kube-api-access-jsvck\") pod \"payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk\" (UID: \"40b5f840-1e90-4f4c-84de-99857ceac633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"
Oct 30 18:58:49.915996 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:49.915970    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"]
Oct 30 18:58:49.920112 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:49.920094    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"
Oct 30 18:58:49.931032 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:49.930910    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-86c150-dockercfg-gdfmk\""
Oct 30 18:58:49.939852 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:49.939827    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"]
Oct 30 18:58:50.066162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:50.065344    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15030cae-5b51-4752-a278-051ab0ca7716-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45\" (UID: \"15030cae-5b51-4752-a278-051ab0ca7716\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"
Oct 30 18:58:50.066162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:50.065402    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctpxm\" (UniqueName: \"kubernetes.io/projected/15030cae-5b51-4752-a278-051ab0ca7716-kube-api-access-ctpxm\") pod \"payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45\" (UID: \"15030cae-5b51-4752-a278-051ab0ca7716\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"
Oct 30 18:58:50.066162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:50.065515    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b5f840-1e90-4f4c-84de-99857ceac633-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk\" (UID: \"40b5f840-1e90-4f4c-84de-99857ceac633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"
Oct 30 18:58:50.066162 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:50.065551    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvck\" (UniqueName: \"kubernetes.io/projected/40b5f840-1e90-4f4c-84de-99857ceac633-kube-api-access-jsvck\") pod \"payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk\" (UID: \"40b5f840-1e90-4f4c-84de-99857ceac633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"
Oct 30 18:58:50.120211 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:50.066900    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b5f840-1e90-4f4c-84de-99857ceac633-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk\" (UID: \"40b5f840-1e90-4f4c-84de-99857ceac633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"
Oct 30 18:58:50.171149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:50.167199    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15030cae-5b51-4752-a278-051ab0ca7716-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45\" (UID: \"15030cae-5b51-4752-a278-051ab0ca7716\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"
Oct 30 18:58:50.171149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:50.167254    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctpxm\" (UniqueName: \"kubernetes.io/projected/15030cae-5b51-4752-a278-051ab0ca7716-kube-api-access-ctpxm\") pod \"payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45\" (UID: \"15030cae-5b51-4752-a278-051ab0ca7716\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"
Oct 30 18:58:50.171149 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:50.168631    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15030cae-5b51-4752-a278-051ab0ca7716-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45\" (UID: \"15030cae-5b51-4752-a278-051ab0ca7716\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"
Oct 30 18:58:50.226828 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:50.226778    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctpxm\" (UniqueName: \"kubernetes.io/projected/15030cae-5b51-4752-a278-051ab0ca7716-kube-api-access-ctpxm\") pod \"payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45\" (UID: \"15030cae-5b51-4752-a278-051ab0ca7716\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"
Oct 30 18:58:50.231554 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:50.231512    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsvck\" (UniqueName: \"kubernetes.io/projected/40b5f840-1e90-4f4c-84de-99857ceac633-kube-api-access-jsvck\") pod \"payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk\" (UID: \"40b5f840-1e90-4f4c-84de-99857ceac633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"
Oct 30 18:58:50.234578 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:50.234551    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"
Oct 30 18:58:50.496197 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:50.487835    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"
Oct 30 18:58:50.569284 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:50.569254    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"]
Oct 30 18:58:50.581185 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:58:50.580008    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15030cae_5b51_4752_a278_051ab0ca7716.slice/crio-144eb77eb8047a9ae9e8dbbc19cc6045df9d947dce976285e5bf1274473f9df9 WatchSource:0}: Error finding container 144eb77eb8047a9ae9e8dbbc19cc6045df9d947dce976285e5bf1274473f9df9: Status 404 returned error can't find the container with id 144eb77eb8047a9ae9e8dbbc19cc6045df9d947dce976285e5bf1274473f9df9
Oct 30 18:58:50.978155 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:50.954576    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"]
Oct 30 18:58:51.243111 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:51.243026    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk" event={"ID":"40b5f840-1e90-4f4c-84de-99857ceac633","Type":"ContainerStarted","Data":"5b1305e24dd833987fd601cd2550e7eb9d1ce367f84dedbac411c26d5c7b1560"}
Oct 30 18:58:51.243111 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:51.243104    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk" event={"ID":"40b5f840-1e90-4f4c-84de-99857ceac633","Type":"ContainerStarted","Data":"7f3d49ffb7c4fd7e90e514b508365dbea2c650318e17e5da1bf9df28a7788ef8"}
Oct 30 18:58:51.243637 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:51.243371    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"
Oct 30 18:58:51.245208 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:51.245180    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45" event={"ID":"15030cae-5b51-4752-a278-051ab0ca7716","Type":"ContainerStarted","Data":"e77b377ad3a4d16931452f058153afa9f199d00a63451fb58d84ff4a179b82b7"}
Oct 30 18:58:51.245353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:51.245226    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45" event={"ID":"15030cae-5b51-4752-a278-051ab0ca7716","Type":"ContainerStarted","Data":"144eb77eb8047a9ae9e8dbbc19cc6045df9d947dce976285e5bf1274473f9df9"}
Oct 30 18:58:51.245353 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:51.245267    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"
Oct 30 18:58:51.271236 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:51.271194    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk" podStartSLOduration=2.271181651 podStartE2EDuration="2.271181651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:51.269875254 +0000 UTC m=+1545.138293583" watchObservedRunningTime="2026-10-30 18:58:51.271181651 +0000 UTC m=+1545.139599981"
Oct 30 18:58:51.297395 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:51.297358    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45" podStartSLOduration=2.297346551 podStartE2EDuration="2.297346551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:51.296560698 +0000 UTC m=+1545.164979028" watchObservedRunningTime="2026-10-30 18:58:51.297346551 +0000 UTC m=+1545.165764881"
Oct 30 18:58:56.148870 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.148835    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"]
Oct 30 18:58:56.153276 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.153256    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.161722 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.161314    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-304f22-openshift-default-dockercfg-mxchf\""
Oct 30 18:58:56.166860 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.166517    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"]
Oct 30 18:58:56.263188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.254214    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-envoy\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.263188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.254323    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-data\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.263188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.254373    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13a85f8c-9244-499b-a12f-37d11fb59cba-istiod-ca-cert\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.263188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.254462    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-workload-certs\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.263188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.254510    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-credential-socket\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.263188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.254563    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-podinfo\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.263188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.254660    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-token\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.263188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.254713    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t4n8\" (UniqueName: \"kubernetes.io/projected/13a85f8c-9244-499b-a12f-37d11fb59cba-kube-api-access-2t4n8\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.263188 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.254746    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-workload-socket\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.366151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.359958    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13a85f8c-9244-499b-a12f-37d11fb59cba-istiod-ca-cert\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.366151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.360067    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-workload-certs\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.366151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.360114    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-credential-socket\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.366151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.360205    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-podinfo\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.366151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.360284    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-token\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.366151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.360323    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t4n8\" (UniqueName: \"kubernetes.io/projected/13a85f8c-9244-499b-a12f-37d11fb59cba-kube-api-access-2t4n8\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.366151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.360353    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-workload-socket\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.366151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.360476    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-envoy\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.366151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.360545    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-data\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.366151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.361007    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-data\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.366151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.361814    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13a85f8c-9244-499b-a12f-37d11fb59cba-istiod-ca-cert\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.366151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.362059    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-workload-certs\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.366151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.362317    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-credential-socket\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.394007 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.392845    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-podinfo\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.394007 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.393702    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-workload-socket\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.410098 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.409457    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-envoy\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.444341 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.444305    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t4n8\" (UniqueName: \"kubernetes.io/projected/13a85f8c-9244-499b-a12f-37d11fb59cba-kube-api-access-2t4n8\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.454765 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.454658    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-token\") pod \"e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") " pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.470835 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.470581    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:56.936740 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.936686    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:58:56.936872 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.936799    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:58:56.936872 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.936848    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:58:56.939843 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:56.939754    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"]
Oct 30 18:58:57.344515 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:57.344468    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv" event={"ID":"13a85f8c-9244-499b-a12f-37d11fb59cba","Type":"ContainerStarted","Data":"048efa7ee353ef701682ffe878dd6679f7618964afceb5ae301a663401b692f4"}
Oct 30 18:58:57.345022 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:57.344545    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv" event={"ID":"13a85f8c-9244-499b-a12f-37d11fb59cba","Type":"ContainerStarted","Data":"7a333bc394ff692d19e7029195d1f2ae2b70b2eb2ddd599ea134ff43a30a36ea"}
Oct 30 18:58:57.375227 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:57.375168    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv" podStartSLOduration=1.375148344 podStartE2EDuration="1.375148344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:57.373575588 +0000 UTC m=+1551.241993919" watchObservedRunningTime="2026-10-30 18:58:57.375148344 +0000 UTC m=+1551.243566675"
Oct 30 18:58:57.471366 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:57.471338    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:57.698412 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:58:57.698322    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:58:58.140756 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.140718    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"]
Oct 30 18:58:58.146042 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.146015    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:58:58.156616 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.156545    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-86c150-serving-cert\""
Oct 30 18:58:58.162698 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.162670    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"]
Oct 30 18:58:58.236037 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.236009    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shk9k\" (UniqueName: \"kubernetes.io/projected/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-kube-api-access-shk9k\") pod \"maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:58:58.236236 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.236156    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:58:58.236296 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.236244    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:58:58.236296 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.236287    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:58:58.341184 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.337595    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shk9k\" (UniqueName: \"kubernetes.io/projected/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-kube-api-access-shk9k\") pod \"maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:58:58.341184 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.337722    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:58:58.341184 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.337769    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:58:58.341184 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.337839    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:58:58.341184 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:58:58.338208    4372 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-serving-cert: secret "maas-api-e2e-shared-a-w4-86c150-serving-cert" not found
Oct 30 18:58:58.341184 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:58:58.338287    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-maas-api-tls podName:125aa336-1adb-4d5c-bc7b-42e1b93d85fe nodeName:}" failed. No retries permitted until 2026-10-30 18:58:58.838263578 +0000 UTC m=+1552.706681889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-maas-api-tls") pod "maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w" (UID: "125aa336-1adb-4d5c-bc7b-42e1b93d85fe") : secret "maas-api-e2e-shared-a-w4-86c150-serving-cert" not found
Oct 30 18:58:58.341184 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.339483    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:58:58.377551 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.373784    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:58:58.380615 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.380214    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shk9k\" (UniqueName: \"kubernetes.io/projected/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-kube-api-access-shk9k\") pod \"maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:58:58.476372 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.476301    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:58.482258 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.482234    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"]
Oct 30 18:58:58.513294 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.513267    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"]
Oct 30 18:58:58.846774 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.846742    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:58:58.849254 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:58.849235    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:58:59.073148 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:59.069192    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:58:59.382916 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:59.382885    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"]
Oct 30 18:58:59.384101 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:58:59.384074    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125aa336_1adb_4d5c_bc7b_42e1b93d85fe.slice/crio-a6c2d6d85fdc575c86c88fa63258a0ff764062f41ba5c94c69e3122514738902 WatchSource:0}: Error finding container a6c2d6d85fdc575c86c88fa63258a0ff764062f41ba5c94c69e3122514738902: Status 404 returned error can't find the container with id a6c2d6d85fdc575c86c88fa63258a0ff764062f41ba5c94c69e3122514738902
Oct 30 18:58:59.385190 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:59.385120    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:58:59.386282 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:58:59.386259    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:59:00.394200 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:00.394163    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w" event={"ID":"125aa336-1adb-4d5c-bc7b-42e1b93d85fe","Type":"ContainerStarted","Data":"460eb81fc6adf56d9a022b607526696744ed9c81385430f6053622bd4e6da868"}
Oct 30 18:59:00.394650 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:00.394233    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w" event={"ID":"125aa336-1adb-4d5c-bc7b-42e1b93d85fe","Type":"ContainerStarted","Data":"a6c2d6d85fdc575c86c88fa63258a0ff764062f41ba5c94c69e3122514738902"}
Oct 30 18:59:00.394650 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:00.394355    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:59:00.421312 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:00.421269    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w" podStartSLOduration=1.96616148 podStartE2EDuration="2.421254827s" totalImagesPullingTime="455.093347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:00.418179061 +0000 UTC m=+1554.286597391" watchObservedRunningTime="2026-10-30 18:59:00.421254827 +0000 UTC m=+1554.289673209"
Oct 30 18:59:02.279875 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:02.279847    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"
Oct 30 18:59:02.280345 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:02.279904    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"
Oct 30 18:59:03.509878 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:03.509202    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w" podUID="82358f53-9e9e-4e7c-a4f9-6dc52c326c35" containerName="payload-processing" containerID="cri-o://9748a6601ae0ef76f082451ad7c5d25d684a9818839ddfc758cf28580f79e1a5" gracePeriod=25
Oct 30 18:59:03.524215 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:03.519825    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb" podUID="4b8cedc2-6b43-4606-b240-6afa12806893" containerName="payload-pre-processing" containerID="cri-o://5cc386b1bfa81755a0553845f3f8f9be0ced506656582cd6a2a928dd5246f72d" gracePeriod=25
Oct 30 18:59:03.807904 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:03.807872    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"
Oct 30 18:59:03.822789 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:03.822765    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82358f53-9e9e-4e7c-a4f9-6dc52c326c35-kube-api-access-c4gjb\" (UniqueName: \"kubernetes.io/projected/82358f53-9e9e-4e7c-a4f9-6dc52c326c35-kube-api-access-c4gjb\") pod \"82358f53-9e9e-4e7c-a4f9-6dc52c326c35\" (UID: \"82358f53-9e9e-4e7c-a4f9-6dc52c326c35\") "
Oct 30 18:59:03.822926 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:03.822868    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82358f53-9e9e-4e7c-a4f9-6dc52c326c35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82358f53-9e9e-4e7c-a4f9-6dc52c326c35-plugins-config-volume\") pod \"82358f53-9e9e-4e7c-a4f9-6dc52c326c35\" (UID: \"82358f53-9e9e-4e7c-a4f9-6dc52c326c35\") "
Oct 30 18:59:03.823526 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:03.823469    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82358f53-9e9e-4e7c-a4f9-6dc52c326c35-plugins-config-volume" pod "82358f53-9e9e-4e7c-a4f9-6dc52c326c35" (UID: "82358f53-9e9e-4e7c-a4f9-6dc52c326c35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:03.825551 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:03.825528    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82358f53-9e9e-4e7c-a4f9-6dc52c326c35-kube-api-access-c4gjb" pod "82358f53-9e9e-4e7c-a4f9-6dc52c326c35" (UID: "82358f53-9e9e-4e7c-a4f9-6dc52c326c35"). InnerVolumeSpecName "kube-api-access-c4gjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:03.861765 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:03.861734    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"
Oct 30 18:59:03.924117 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:03.924087    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8cedc2-6b43-4606-b240-6afa12806893-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8cedc2-6b43-4606-b240-6afa12806893-plugins-config-volume\") pod \"4b8cedc2-6b43-4606-b240-6afa12806893\" (UID: \"4b8cedc2-6b43-4606-b240-6afa12806893\") "
Oct 30 18:59:03.924308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:03.924170    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8cedc2-6b43-4606-b240-6afa12806893-kube-api-access-sk9kd\" (UniqueName: \"kubernetes.io/projected/4b8cedc2-6b43-4606-b240-6afa12806893-kube-api-access-sk9kd\") pod \"4b8cedc2-6b43-4606-b240-6afa12806893\" (UID: \"4b8cedc2-6b43-4606-b240-6afa12806893\") "
Oct 30 18:59:03.924695 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:03.924665    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8cedc2-6b43-4606-b240-6afa12806893-plugins-config-volume" pod "4b8cedc2-6b43-4606-b240-6afa12806893" (UID: "4b8cedc2-6b43-4606-b240-6afa12806893"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:03.925204 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:03.925180    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82358f53-9e9e-4e7c-a4f9-6dc52c326c35-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:03.925265 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:03.925211    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8cedc2-6b43-4606-b240-6afa12806893-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:03.925265 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:03.925226    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4gjb\" (UniqueName: \"kubernetes.io/projected/82358f53-9e9e-4e7c-a4f9-6dc52c326c35-kube-api-access-c4gjb\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:03.926620 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:03.926599    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8cedc2-6b43-4606-b240-6afa12806893-kube-api-access-sk9kd" pod "4b8cedc2-6b43-4606-b240-6afa12806893" (UID: "4b8cedc2-6b43-4606-b240-6afa12806893"). InnerVolumeSpecName "kube-api-access-sk9kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:04.026553 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.026525    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk9kd\" (UniqueName: \"kubernetes.io/projected/4b8cedc2-6b43-4606-b240-6afa12806893-kube-api-access-sk9kd\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:04.421100 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.421066    4372 generic.go:356] "Generic (PLEG): container finished" podID="4b8cedc2-6b43-4606-b240-6afa12806893" containerID="5cc386b1bfa81755a0553845f3f8f9be0ced506656582cd6a2a928dd5246f72d" exitCode=0
Oct 30 18:59:04.421328 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.421139    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"
Oct 30 18:59:04.421328 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.421160    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb" event={"ID":"4b8cedc2-6b43-4606-b240-6afa12806893","Type":"ContainerDied","Data":"5cc386b1bfa81755a0553845f3f8f9be0ced506656582cd6a2a928dd5246f72d"}
Oct 30 18:59:04.421328 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.421221    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb" event={"ID":"4b8cedc2-6b43-4606-b240-6afa12806893","Type":"ContainerDied","Data":"cb139395a4c73c069bfc4c1d7e9e565d32336ffce050d42ac7c1df1d658effbd"}
Oct 30 18:59:04.421328 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.421253    4372 scope.go:122] "RemoveContainer" containerID="5cc386b1bfa81755a0553845f3f8f9be0ced506656582cd6a2a928dd5246f72d"
Oct 30 18:59:04.422694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.422668    4372 generic.go:356] "Generic (PLEG): container finished" podID="82358f53-9e9e-4e7c-a4f9-6dc52c326c35" containerID="9748a6601ae0ef76f082451ad7c5d25d684a9818839ddfc758cf28580f79e1a5" exitCode=0
Oct 30 18:59:04.422845 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.422741    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"
Oct 30 18:59:04.422845 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.422756    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w" event={"ID":"82358f53-9e9e-4e7c-a4f9-6dc52c326c35","Type":"ContainerDied","Data":"9748a6601ae0ef76f082451ad7c5d25d684a9818839ddfc758cf28580f79e1a5"}
Oct 30 18:59:04.422969 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.422841    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w" event={"ID":"82358f53-9e9e-4e7c-a4f9-6dc52c326c35","Type":"ContainerDied","Data":"a485d7b039344e9aaf5c56e21503ecd729ca6f45387ba853fc0d7c188a0ba115"}
Oct 30 18:59:04.431617 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.431582    4372 scope.go:122] "RemoveContainer" containerID="5cc386b1bfa81755a0553845f3f8f9be0ced506656582cd6a2a928dd5246f72d"
Oct 30 18:59:04.431904 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:59:04.431882    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc386b1bfa81755a0553845f3f8f9be0ced506656582cd6a2a928dd5246f72d\": container with ID starting with 5cc386b1bfa81755a0553845f3f8f9be0ced506656582cd6a2a928dd5246f72d not found: ID does not exist" containerID="5cc386b1bfa81755a0553845f3f8f9be0ced506656582cd6a2a928dd5246f72d"
Oct 30 18:59:04.432017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.431914    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc386b1bfa81755a0553845f3f8f9be0ced506656582cd6a2a928dd5246f72d"} err="failed to get container status \"5cc386b1bfa81755a0553845f3f8f9be0ced506656582cd6a2a928dd5246f72d\": rpc error: code = NotFound desc = could not find container \"5cc386b1bfa81755a0553845f3f8f9be0ced506656582cd6a2a928dd5246f72d\": container with ID starting with 5cc386b1bfa81755a0553845f3f8f9be0ced506656582cd6a2a928dd5246f72d not found: ID does not exist"
Oct 30 18:59:04.432017 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.431936    4372 scope.go:122] "RemoveContainer" containerID="9748a6601ae0ef76f082451ad7c5d25d684a9818839ddfc758cf28580f79e1a5"
Oct 30 18:59:04.442338 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.442320    4372 scope.go:122] "RemoveContainer" containerID="9748a6601ae0ef76f082451ad7c5d25d684a9818839ddfc758cf28580f79e1a5"
Oct 30 18:59:04.442620 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:59:04.442600    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9748a6601ae0ef76f082451ad7c5d25d684a9818839ddfc758cf28580f79e1a5\": container with ID starting with 9748a6601ae0ef76f082451ad7c5d25d684a9818839ddfc758cf28580f79e1a5 not found: ID does not exist" containerID="9748a6601ae0ef76f082451ad7c5d25d684a9818839ddfc758cf28580f79e1a5"
Oct 30 18:59:04.442682 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.442625    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9748a6601ae0ef76f082451ad7c5d25d684a9818839ddfc758cf28580f79e1a5"} err="failed to get container status \"9748a6601ae0ef76f082451ad7c5d25d684a9818839ddfc758cf28580f79e1a5\": rpc error: code = NotFound desc = could not find container \"9748a6601ae0ef76f082451ad7c5d25d684a9818839ddfc758cf28580f79e1a5\": container with ID starting with 9748a6601ae0ef76f082451ad7c5d25d684a9818839ddfc758cf28580f79e1a5 not found: ID does not exist"
Oct 30 18:59:04.445842 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.445817    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"]
Oct 30 18:59:04.447788 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.447755    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fc28ee-557d45596c-v4bqb"]
Oct 30 18:59:04.462037 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.462018    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"]
Oct 30 18:59:04.467277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:04.467255    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fc28ee-6cbcb57bcd-nww8w"]
Oct 30 18:59:06.062683 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.062646    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"]
Oct 30 18:59:06.063245 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.062893    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5" podUID="0ea33820-3049-447c-9369-6dd72bd8c85d" containerName="maas-api" containerID="cri-o://780ceb3f28a06c44f4ba360f466a26a2aa21b3a22985add4642c923ace928a6f" gracePeriod=30
Oct 30 18:59:06.269663 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.269629    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8cedc2-6b43-4606-b240-6afa12806893" path="/var/lib/kubelet/pods/4b8cedc2-6b43-4606-b240-6afa12806893/volumes"
Oct 30 18:59:06.270232 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.270212    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82358f53-9e9e-4e7c-a4f9-6dc52c326c35" path="/var/lib/kubelet/pods/82358f53-9e9e-4e7c-a4f9-6dc52c326c35/volumes"
Oct 30 18:59:06.431220 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.422317    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:59:06.431220 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.429744    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 18:59:06.480350 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.480312    4372 generic.go:356] "Generic (PLEG): container finished" podID="0ea33820-3049-447c-9369-6dd72bd8c85d" containerID="780ceb3f28a06c44f4ba360f466a26a2aa21b3a22985add4642c923ace928a6f" exitCode=0
Oct 30 18:59:06.480547 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.480377    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5" event={"ID":"0ea33820-3049-447c-9369-6dd72bd8c85d","Type":"ContainerDied","Data":"780ceb3f28a06c44f4ba360f466a26a2aa21b3a22985add4642c923ace928a6f"}
Oct 30 18:59:06.480547 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.480424    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5" event={"ID":"0ea33820-3049-447c-9369-6dd72bd8c85d","Type":"ContainerDied","Data":"11a5d25bd7a7cf2456b37559d68831ad5fae26677919760ec746e765ad11ba00"}
Oct 30 18:59:06.480547 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.480451    4372 scope.go:122] "RemoveContainer" containerID="780ceb3f28a06c44f4ba360f466a26a2aa21b3a22985add4642c923ace928a6f"
Oct 30 18:59:06.480702 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.480559    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"
Oct 30 18:59:06.556486 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.556039    4372 scope.go:122] "RemoveContainer" containerID="780ceb3f28a06c44f4ba360f466a26a2aa21b3a22985add4642c923ace928a6f"
Oct 30 18:59:06.556587 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:59:06.556499    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"780ceb3f28a06c44f4ba360f466a26a2aa21b3a22985add4642c923ace928a6f\": container with ID starting with 780ceb3f28a06c44f4ba360f466a26a2aa21b3a22985add4642c923ace928a6f not found: ID does not exist" containerID="780ceb3f28a06c44f4ba360f466a26a2aa21b3a22985add4642c923ace928a6f"
Oct 30 18:59:06.556587 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.556538    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780ceb3f28a06c44f4ba360f466a26a2aa21b3a22985add4642c923ace928a6f"} err="failed to get container status \"780ceb3f28a06c44f4ba360f466a26a2aa21b3a22985add4642c923ace928a6f\": rpc error: code = NotFound desc = could not find container \"780ceb3f28a06c44f4ba360f466a26a2aa21b3a22985add4642c923ace928a6f\": container with ID starting with 780ceb3f28a06c44f4ba360f466a26a2aa21b3a22985add4642c923ace928a6f not found: ID does not exist"
Oct 30 18:59:06.590788 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.590763    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ea33820-3049-447c-9369-6dd72bd8c85d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea33820-3049-447c-9369-6dd72bd8c85d-openshift-service-ca\") pod \"0ea33820-3049-447c-9369-6dd72bd8c85d\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") "
Oct 30 18:59:06.590939 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.590852    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-maas-api-tls\") pod \"0ea33820-3049-447c-9369-6dd72bd8c85d\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") "
Oct 30 18:59:06.590939 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.590915    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-metrics-certs\") pod \"0ea33820-3049-447c-9369-6dd72bd8c85d\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") "
Oct 30 18:59:06.591028 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.591011    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea33820-3049-447c-9369-6dd72bd8c85d-kube-api-access-v7f84\" (UniqueName: \"kubernetes.io/projected/0ea33820-3049-447c-9369-6dd72bd8c85d-kube-api-access-v7f84\") pod \"0ea33820-3049-447c-9369-6dd72bd8c85d\" (UID: \"0ea33820-3049-447c-9369-6dd72bd8c85d\") "
Oct 30 18:59:06.591111 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.591088    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea33820-3049-447c-9369-6dd72bd8c85d-openshift-service-ca" pod "0ea33820-3049-447c-9369-6dd72bd8c85d" (UID: "0ea33820-3049-447c-9369-6dd72bd8c85d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:06.591718 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.591691    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea33820-3049-447c-9369-6dd72bd8c85d-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:06.593234 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.593203    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-metrics-certs" pod "0ea33820-3049-447c-9369-6dd72bd8c85d" (UID: "0ea33820-3049-447c-9369-6dd72bd8c85d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:06.593349 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.593279    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-maas-api-tls" pod "0ea33820-3049-447c-9369-6dd72bd8c85d" (UID: "0ea33820-3049-447c-9369-6dd72bd8c85d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:06.593349 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.593341    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea33820-3049-447c-9369-6dd72bd8c85d-kube-api-access-v7f84" pod "0ea33820-3049-447c-9369-6dd72bd8c85d" (UID: "0ea33820-3049-447c-9369-6dd72bd8c85d"). InnerVolumeSpecName "kube-api-access-v7f84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:06.693273 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.693202    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:06.693273 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.693228    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ea33820-3049-447c-9369-6dd72bd8c85d-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:06.693273 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.693239    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7f84\" (UniqueName: \"kubernetes.io/projected/0ea33820-3049-447c-9369-6dd72bd8c85d-kube-api-access-v7f84\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:06.827948 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.827913    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"]
Oct 30 18:59:06.832173 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.832143    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fc28ee-75485b8c96-mn4d5"]
Oct 30 18:59:06.881311 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:06.881274    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:59:08.296152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:08.292911    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea33820-3049-447c-9369-6dd72bd8c85d" path="/var/lib/kubelet/pods/0ea33820-3049-447c-9369-6dd72bd8c85d/volumes"
Oct 30 18:59:09.586254 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:09.586202    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 18:59:09.899521 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:09.899396    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:09.900337 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:09.900287    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 18:59:09.902834 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:09.902787    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 18:59:09.917102 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:09.916208    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:10.900220 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:10.900177    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"]
Oct 30 18:59:10.925196 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:10.901719    4372 cpu_manager.go:414] "containerMap: removing container" podUID="0ea33820-3049-447c-9369-6dd72bd8c85d" containerName="maas-api"
Oct 30 18:59:10.940202 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:10.940166    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea33820-3049-447c-9369-6dd72bd8c85d" containerName="maas-api"
Oct 30 18:59:10.940514 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:10.940494    4372 cpu_manager.go:414] "containerMap: removing container" podUID="4b8cedc2-6b43-4606-b240-6afa12806893" containerName="payload-pre-processing"
Oct 30 18:59:10.940649 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:10.940632    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8cedc2-6b43-4606-b240-6afa12806893" containerName="payload-pre-processing"
Oct 30 18:59:10.955150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:10.940773    4372 cpu_manager.go:414] "containerMap: removing container" podUID="82358f53-9e9e-4e7c-a4f9-6dc52c326c35" containerName="payload-processing"
Oct 30 18:59:10.958921 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:10.955736    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82358f53-9e9e-4e7c-a4f9-6dc52c326c35" containerName="payload-processing"
Oct 30 18:59:10.958921 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:10.955990    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea33820-3049-447c-9369-6dd72bd8c85d" containerName="maas-api"
Oct 30 18:59:10.958921 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:10.956019    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8cedc2-6b43-4606-b240-6afa12806893" containerName="payload-pre-processing"
Oct 30 18:59:10.958921 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:10.956032    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="82358f53-9e9e-4e7c-a4f9-6dc52c326c35" containerName="payload-processing"
Oct 30 18:59:11.025157 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:10.999978    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"]
Oct 30 18:59:11.025418 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:11.008642    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:11.042044 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:11.036852    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-ab7df0-serving-cert\""
Oct 30 18:59:11.106214 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:11.104822    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18722d33-fb04-43df-bf56-3722c86e6346-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:11.106214 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:11.105026    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18722d33-fb04-43df-bf56-3722c86e6346-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:11.106214 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:11.105412    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18722d33-fb04-43df-bf56-3722c86e6346-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:11.106214 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:11.105625    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk5mj\" (UniqueName: \"kubernetes.io/projected/18722d33-fb04-43df-bf56-3722c86e6346-kube-api-access-jk5mj\") pod \"maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:11.214984 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:11.209461    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18722d33-fb04-43df-bf56-3722c86e6346-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:11.214984 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:11.209526    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk5mj\" (UniqueName: \"kubernetes.io/projected/18722d33-fb04-43df-bf56-3722c86e6346-kube-api-access-jk5mj\") pod \"maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:11.214984 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:11.209681    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18722d33-fb04-43df-bf56-3722c86e6346-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:11.214984 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:11.209876    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18722d33-fb04-43df-bf56-3722c86e6346-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:11.214984 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:11.211409    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18722d33-fb04-43df-bf56-3722c86e6346-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:11.240263 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:11.238560    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18722d33-fb04-43df-bf56-3722c86e6346-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:11.258318 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:11.244761    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk5mj\" (UniqueName: \"kubernetes.io/projected/18722d33-fb04-43df-bf56-3722c86e6346-kube-api-access-jk5mj\") pod \"maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:11.261269 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:11.261244    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18722d33-fb04-43df-bf56-3722c86e6346-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:11.409152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:11.406380    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:12.025363 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.025274    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"]
Oct 30 18:59:12.031825 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.031800    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:12.039211 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.038723    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-304f22-serving-cert\""
Oct 30 18:59:12.041274 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.041176    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"]
Oct 30 18:59:12.062989 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.062968    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"]
Oct 30 18:59:12.064600 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:59:12.064564    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18722d33_fb04_43df_bf56_3722c86e6346.slice/crio-d5430e9c190ddf577a9d7fdebd16e6e6feeca2626e33e4d0c466c1894c826caf WatchSource:0}: Error finding container d5430e9c190ddf577a9d7fdebd16e6e6feeca2626e33e4d0c466c1894c826caf: Status 404 returned error can't find the container with id d5430e9c190ddf577a9d7fdebd16e6e6feeca2626e33e4d0c466c1894c826caf
Oct 30 18:59:12.066435 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.066386    4372 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:59:12.131677 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.130522    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:12.131677 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.130770    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhdsx\" (UniqueName: \"kubernetes.io/projected/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-kube-api-access-dhdsx\") pod \"maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:12.131677 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.131214    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:12.131677 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.131370    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:12.240167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.237955    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhdsx\" (UniqueName: \"kubernetes.io/projected/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-kube-api-access-dhdsx\") pod \"maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:12.240167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.238286    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:12.240167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.238415    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:12.240167 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.238552    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:12.257156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.255849    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:12.285682 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.285570    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:12.295496 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.286766    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:12.321335 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.321302    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhdsx\" (UniqueName: \"kubernetes.io/projected/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-kube-api-access-dhdsx\") pod \"maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:12.352254 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.352219    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:12.622798 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.622763    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh" event={"ID":"18722d33-fb04-43df-bf56-3722c86e6346","Type":"ContainerStarted","Data":"d5430e9c190ddf577a9d7fdebd16e6e6feeca2626e33e4d0c466c1894c826caf"}
Oct 30 18:59:12.675896 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:59:12.675872    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd812973_9a0c_43a0_8bf2_982c4ef59bd2.slice/crio-def3dfb65017bb7eed89dd4ae733f0aca894662e9b908d62f9060eac6f890ac9 WatchSource:0}: Error finding container def3dfb65017bb7eed89dd4ae733f0aca894662e9b908d62f9060eac6f890ac9: Status 404 returned error can't find the container with id def3dfb65017bb7eed89dd4ae733f0aca894662e9b908d62f9060eac6f890ac9
Oct 30 18:59:12.676286 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:12.676265    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"]
Oct 30 18:59:13.632638 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:13.632593    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh" event={"ID":"18722d33-fb04-43df-bf56-3722c86e6346","Type":"ContainerStarted","Data":"89a4134d2acb27d824964d360250450948d131baf8768f97624c33757806c4c2"}
Oct 30 18:59:13.633144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:13.632790    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:13.634390 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:13.634369    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs" event={"ID":"bd812973-9a0c-43a0-8bf2-982c4ef59bd2","Type":"ContainerStarted","Data":"5dfcda1a916ea385fb5edae213a6860a1a2cecf2db9a9d072f1064a309b5a3de"}
Oct 30 18:59:13.634493 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:13.634403    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs" event={"ID":"bd812973-9a0c-43a0-8bf2-982c4ef59bd2","Type":"ContainerStarted","Data":"def3dfb65017bb7eed89dd4ae733f0aca894662e9b908d62f9060eac6f890ac9"}
Oct 30 18:59:13.634536 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:13.634497    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:13.659182 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:13.659142    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh" podStartSLOduration=3.091346522 podStartE2EDuration="3.659117519s" totalImagesPullingTime="567.770997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:13.655851494 +0000 UTC m=+1567.524269825" watchObservedRunningTime="2026-10-30 18:59:13.659117519 +0000 UTC m=+1567.527535871"
Oct 30 18:59:13.707068 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:13.707010    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs" podStartSLOduration=2.2168354 podStartE2EDuration="2.70699489s" totalImagesPullingTime="490.15949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:13.70669867 +0000 UTC m=+1567.575117001" watchObservedRunningTime="2026-10-30 18:59:13.70699489 +0000 UTC m=+1567.575413222"
Oct 30 18:59:16.216091 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.216056    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"]
Oct 30 18:59:16.222160 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.222077    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.230446 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.230012    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-c3a5b9-openshift-default-dockercfg-b8nm9\""
Oct 30 18:59:16.236512 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.236488    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"]
Oct 30 18:59:16.261450 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.261344    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-token\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.261450 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.261405    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-workload-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.261941 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.261799    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36220b42-54dc-4dd2-aac8-e44b93f61a55-istiod-ca-cert\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.261941 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.261844    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6wr9\" (UniqueName: \"kubernetes.io/projected/36220b42-54dc-4dd2-aac8-e44b93f61a55-kube-api-access-b6wr9\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.261941 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.261915    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-podinfo\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.262144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.261968    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-credential-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.262144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.262019    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-workload-certs\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.262144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.262047    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-data\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.262144 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.262088    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-envoy\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.370352 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.366647    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-credential-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.370352 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.366725    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-workload-certs\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.370352 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.366762    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-data\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.370352 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.366803    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-envoy\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.392726 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.392694    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-token\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.393020 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.392995    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-workload-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.401154 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.393241    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36220b42-54dc-4dd2-aac8-e44b93f61a55-istiod-ca-cert\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.401384 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.401358    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6wr9\" (UniqueName: \"kubernetes.io/projected/36220b42-54dc-4dd2-aac8-e44b93f61a55-kube-api-access-b6wr9\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.401629 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.401611    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-podinfo\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.425153 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.395323    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-data\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.426352 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.415462    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-envoy\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.426542 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.395031    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-workload-certs\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.426644 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.415708    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-workload-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.426742 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.394780    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-credential-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.436511 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.436299    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36220b42-54dc-4dd2-aac8-e44b93f61a55-istiod-ca-cert\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.473107 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.472621    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-token\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.512151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.511639    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6wr9\" (UniqueName: \"kubernetes.io/projected/36220b42-54dc-4dd2-aac8-e44b93f61a55-kube-api-access-b6wr9\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.523479 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.521020    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-podinfo\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.542959 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.542674    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:16.984303 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.984273    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"]
Oct 30 18:59:16.984625 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:59:16.984594    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36220b42_54dc_4dd2_aac8_e44b93f61a55.slice/crio-7aee46a91d6c1c138ceaddebbe76eb5461f53ea682560081d57dfe22717f0f0b WatchSource:0}: Error finding container 7aee46a91d6c1c138ceaddebbe76eb5461f53ea682560081d57dfe22717f0f0b: Status 404 returned error can't find the container with id 7aee46a91d6c1c138ceaddebbe76eb5461f53ea682560081d57dfe22717f0f0b
Oct 30 18:59:16.988609 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.988560    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:59:16.988725 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.988657    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:59:16.988725 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:16.988703    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:59:17.734369 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:17.734328    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd" event={"ID":"36220b42-54dc-4dd2-aac8-e44b93f61a55","Type":"ContainerStarted","Data":"dccce248504137e4bd9e5c375b4d844071c7f1692e23f9eb75e1c57e7b1b4ecb"}
Oct 30 18:59:17.734856 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:17.734393    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd" event={"ID":"36220b42-54dc-4dd2-aac8-e44b93f61a55","Type":"ContainerStarted","Data":"7aee46a91d6c1c138ceaddebbe76eb5461f53ea682560081d57dfe22717f0f0b"}
Oct 30 18:59:17.765417 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:17.765354    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd" podStartSLOduration=1.765334065 podStartE2EDuration="1.765334065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:17.760143712 +0000 UTC m=+1571.628562033" watchObservedRunningTime="2026-10-30 18:59:17.765334065 +0000 UTC m=+1571.633752386"
Oct 30 18:59:18.543358 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:18.543330    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:18.548877 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:18.548854    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:18.743801 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:18.743247    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:18.744531 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:18.744508    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 18:59:19.692401 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:19.692358    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 18:59:19.701351 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:19.701302    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 18:59:20.824150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:20.823359    4372 generic.go:356] "Generic (PLEG): container finished" podID="852083d5-0743-44c0-85fd-0551043be8d7" containerID="9e2aa3981d9dff77a42096c37cbc58c4a7c277f468b803c38a4822f1baf8ca9e" exitCode=0
Oct 30 18:59:20.824150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:20.824090    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv" event={"ID":"852083d5-0743-44c0-85fd-0551043be8d7","Type":"ContainerDied","Data":"9e2aa3981d9dff77a42096c37cbc58c4a7c277f468b803c38a4822f1baf8ca9e"}
Oct 30 18:59:20.824781 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:20.824566    4372 scope.go:122] "RemoveContainer" containerID="9e2aa3981d9dff77a42096c37cbc58c4a7c277f468b803c38a4822f1baf8ca9e"
Oct 30 18:59:20.826105 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:20.826073    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:59:20.826231 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:20.826172    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:59:20.826231 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:20.826224    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:59:21.834652 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:21.834618    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv" event={"ID":"852083d5-0743-44c0-85fd-0551043be8d7","Type":"ContainerStarted","Data":"1114d1ae08ad231b03e11a64dc39e3e32071b17c2483c73b41b7b17873318eca"}
Oct 30 18:59:22.128203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:22.117764    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:59:22.128203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:22.117886    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 18:59:22.128203 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:22.117929    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv" podUID="852083d5-0743-44c0-85fd-0551043be8d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 18:59:23.114667 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:23.114617    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 18:59:23.115058 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:23.114687    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv" podUID="852083d5-0743-44c0-85fd-0551043be8d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 18:59:24.114916 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:24.114873    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 18:59:24.115454 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:24.114959    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv" podUID="852083d5-0743-44c0-85fd-0551043be8d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 18:59:25.114004 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:25.113969    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 18:59:25.114230 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:25.114019    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv" podUID="852083d5-0743-44c0-85fd-0551043be8d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 18:59:26.114619 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:26.114577    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 18:59:26.115027 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:26.114634    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv" podUID="852083d5-0743-44c0-85fd-0551043be8d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 18:59:26.287151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:26.278600    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"]
Oct 30 18:59:26.317245 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:26.317192    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"]
Oct 30 18:59:27.114543 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:27.114507    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 18:59:27.114730 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:27.114560    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv" podUID="852083d5-0743-44c0-85fd-0551043be8d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 18:59:27.696151 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:59:27.696084    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:59:27.924439 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:27.924408    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:59:28.118887 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:28.118861    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:59:28.886530 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:28.886498    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 18:59:31.279902 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:31.279844    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z" podUID="6f308cee-8550-48a4-b417-0f49cde47af8" containerName="payload-processing" containerID="cri-o://28fbbc65c9af3f5be9e47ecc385a9edee27113566d6f9677ea4a4acbb8b68108" gracePeriod=25
Oct 30 18:59:31.319246 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:31.319206    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6" podUID="8d81ca16-a936-4c2b-adee-174cdc1e3266" containerName="payload-pre-processing" containerID="cri-o://4dfe130bb88fd0e0e3f0a6ab1ca596317fcbf3c136b9438474d10a6c83b7c8ee" gracePeriod=25
Oct 30 18:59:31.928205 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:31.915796    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"
Oct 30 18:59:31.965732 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:31.965696    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"
Oct 30 18:59:32.026174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.019166    4372 generic.go:356] "Generic (PLEG): container finished" podID="8d81ca16-a936-4c2b-adee-174cdc1e3266" containerID="4dfe130bb88fd0e0e3f0a6ab1ca596317fcbf3c136b9438474d10a6c83b7c8ee" exitCode=0
Oct 30 18:59:32.026174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.019301    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6" event={"ID":"8d81ca16-a936-4c2b-adee-174cdc1e3266","Type":"ContainerDied","Data":"4dfe130bb88fd0e0e3f0a6ab1ca596317fcbf3c136b9438474d10a6c83b7c8ee"}
Oct 30 18:59:32.026174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.019396    4372 scope.go:122] "RemoveContainer" containerID="4dfe130bb88fd0e0e3f0a6ab1ca596317fcbf3c136b9438474d10a6c83b7c8ee"
Oct 30 18:59:32.026174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.019612    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"
Oct 30 18:59:32.026174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.023064    4372 generic.go:356] "Generic (PLEG): container finished" podID="6f308cee-8550-48a4-b417-0f49cde47af8" containerID="28fbbc65c9af3f5be9e47ecc385a9edee27113566d6f9677ea4a4acbb8b68108" exitCode=0
Oct 30 18:59:32.026174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.023305    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z" event={"ID":"6f308cee-8550-48a4-b417-0f49cde47af8","Type":"ContainerDied","Data":"28fbbc65c9af3f5be9e47ecc385a9edee27113566d6f9677ea4a4acbb8b68108"}
Oct 30 18:59:32.026174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.023356    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z" event={"ID":"6f308cee-8550-48a4-b417-0f49cde47af8","Type":"ContainerDied","Data":"598d8e00464536690194fac17d659347d1c4efbef2f4556ef7e5a59a35957b35"}
Oct 30 18:59:32.026174 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.023460    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"
Oct 30 18:59:32.075152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.048773    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d81ca16-a936-4c2b-adee-174cdc1e3266-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d81ca16-a936-4c2b-adee-174cdc1e3266-plugins-config-volume\") pod \"8d81ca16-a936-4c2b-adee-174cdc1e3266\" (UID: \"8d81ca16-a936-4c2b-adee-174cdc1e3266\") "
Oct 30 18:59:32.075152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.048835    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d81ca16-a936-4c2b-adee-174cdc1e3266-kube-api-access-rfr7q\" (UniqueName: \"kubernetes.io/projected/8d81ca16-a936-4c2b-adee-174cdc1e3266-kube-api-access-rfr7q\") pod \"8d81ca16-a936-4c2b-adee-174cdc1e3266\" (UID: \"8d81ca16-a936-4c2b-adee-174cdc1e3266\") "
Oct 30 18:59:32.075152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.049003    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f308cee-8550-48a4-b417-0f49cde47af8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f308cee-8550-48a4-b417-0f49cde47af8-plugins-config-volume\") pod \"6f308cee-8550-48a4-b417-0f49cde47af8\" (UID: \"6f308cee-8550-48a4-b417-0f49cde47af8\") "
Oct 30 18:59:32.075152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.049069    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f308cee-8550-48a4-b417-0f49cde47af8-kube-api-access-b7zzk\" (UniqueName: \"kubernetes.io/projected/6f308cee-8550-48a4-b417-0f49cde47af8-kube-api-access-b7zzk\") pod \"6f308cee-8550-48a4-b417-0f49cde47af8\" (UID: \"6f308cee-8550-48a4-b417-0f49cde47af8\") "
Oct 30 18:59:32.075152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.050850    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d81ca16-a936-4c2b-adee-174cdc1e3266-plugins-config-volume" pod "8d81ca16-a936-4c2b-adee-174cdc1e3266" (UID: "8d81ca16-a936-4c2b-adee-174cdc1e3266"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:32.075152 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.051266    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f308cee-8550-48a4-b417-0f49cde47af8-plugins-config-volume" pod "6f308cee-8550-48a4-b417-0f49cde47af8" (UID: "6f308cee-8550-48a4-b417-0f49cde47af8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:32.131404 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.127529    4372 scope.go:122] "RemoveContainer" containerID="28fbbc65c9af3f5be9e47ecc385a9edee27113566d6f9677ea4a4acbb8b68108"
Oct 30 18:59:32.131404 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.129434    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f308cee-8550-48a4-b417-0f49cde47af8-kube-api-access-b7zzk" pod "6f308cee-8550-48a4-b417-0f49cde47af8" (UID: "6f308cee-8550-48a4-b417-0f49cde47af8"). InnerVolumeSpecName "kube-api-access-b7zzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:32.167379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.156317    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d81ca16-a936-4c2b-adee-174cdc1e3266-kube-api-access-rfr7q" pod "8d81ca16-a936-4c2b-adee-174cdc1e3266" (UID: "8d81ca16-a936-4c2b-adee-174cdc1e3266"). InnerVolumeSpecName "kube-api-access-rfr7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:32.167379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.156599    4372 scope.go:122] "RemoveContainer" containerID="28fbbc65c9af3f5be9e47ecc385a9edee27113566d6f9677ea4a4acbb8b68108"
Oct 30 18:59:32.167379 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:59:32.156973    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fbbc65c9af3f5be9e47ecc385a9edee27113566d6f9677ea4a4acbb8b68108\": container with ID starting with 28fbbc65c9af3f5be9e47ecc385a9edee27113566d6f9677ea4a4acbb8b68108 not found: ID does not exist" containerID="28fbbc65c9af3f5be9e47ecc385a9edee27113566d6f9677ea4a4acbb8b68108"
Oct 30 18:59:32.167379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.157007    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fbbc65c9af3f5be9e47ecc385a9edee27113566d6f9677ea4a4acbb8b68108"} err="failed to get container status \"28fbbc65c9af3f5be9e47ecc385a9edee27113566d6f9677ea4a4acbb8b68108\": rpc error: code = NotFound desc = could not find container \"28fbbc65c9af3f5be9e47ecc385a9edee27113566d6f9677ea4a4acbb8b68108\": container with ID starting with 28fbbc65c9af3f5be9e47ecc385a9edee27113566d6f9677ea4a4acbb8b68108 not found: ID does not exist"
Oct 30 18:59:32.167379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.157310    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f308cee-8550-48a4-b417-0f49cde47af8-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:32.167379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.157329    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7zzk\" (UniqueName: \"kubernetes.io/projected/6f308cee-8550-48a4-b417-0f49cde47af8-kube-api-access-b7zzk\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:32.167379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.157340    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d81ca16-a936-4c2b-adee-174cdc1e3266-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:32.167379 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.157349    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfr7q\" (UniqueName: \"kubernetes.io/projected/8d81ca16-a936-4c2b-adee-174cdc1e3266-kube-api-access-rfr7q\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:32.371002 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.370964    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"]
Oct 30 18:59:32.377588 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.377564    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bcb5af-b5b6c4b8-lzd8z"]
Oct 30 18:59:32.433070 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.430261    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"]
Oct 30 18:59:32.436248 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.436069    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bcb5af-68968d8455-zgnf6"]
Oct 30 18:59:32.456308 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.456280    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"]
Oct 30 18:59:32.456887 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.456871    4372 cpu_manager.go:414] "containerMap: removing container" podUID="8d81ca16-a936-4c2b-adee-174cdc1e3266" containerName="payload-pre-processing"
Oct 30 18:59:32.456978 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.456888    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d81ca16-a936-4c2b-adee-174cdc1e3266" containerName="payload-pre-processing"
Oct 30 18:59:32.456978 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.456919    4372 cpu_manager.go:414] "containerMap: removing container" podUID="6f308cee-8550-48a4-b417-0f49cde47af8" containerName="payload-processing"
Oct 30 18:59:32.456978 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.456925    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f308cee-8550-48a4-b417-0f49cde47af8" containerName="payload-processing"
Oct 30 18:59:32.457145 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.457021    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f308cee-8550-48a4-b417-0f49cde47af8" containerName="payload-processing"
Oct 30 18:59:32.457145 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.457036    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d81ca16-a936-4c2b-adee-174cdc1e3266" containerName="payload-pre-processing"
Oct 30 18:59:32.461874 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.461857    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.470316 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.470293    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a962a\"/\"openshift-service-ca.crt\""
Oct 30 18:59:32.470497 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.470477    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a962a\"/\"kube-root-ca.crt\""
Oct 30 18:59:32.472172 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.472058    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"]
Oct 30 18:59:32.562686 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.562599    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-home\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.562840 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.562783    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-dshm\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.562888 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.562851    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c97b261-f060-45a3-95e5-a6f924b51047-tls-certs\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.562940 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.562923    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-tmp-dir\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.562981 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.562959    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-654ht\" (UniqueName: \"kubernetes.io/projected/3c97b261-f060-45a3-95e5-a6f924b51047-kube-api-access-654ht\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.563107 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.563085    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-model-cache\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.566957 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.566939    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a962a\"/\"auth-test-model-0a962a-kserve-self-signed-certs\""
Oct 30 18:59:32.567019 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.566959    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a962a\"/\"default-dockercfg-r6vph\""
Oct 30 18:59:32.664725 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.664689    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c97b261-f060-45a3-95e5-a6f924b51047-tls-certs\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.664896 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.664753    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-tmp-dir\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.664896 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.664795    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-654ht\" (UniqueName: \"kubernetes.io/projected/3c97b261-f060-45a3-95e5-a6f924b51047-kube-api-access-654ht\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.665050 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.664899    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-model-cache\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.665050 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.664944    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-home\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.665196 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.665097    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-dshm\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.665196 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.665181    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-tmp-dir\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.665340 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.665319    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-home\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.665675 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.665657    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-model-cache\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.667416 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.667391    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-dshm\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.667549 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.667400    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c97b261-f060-45a3-95e5-a6f924b51047-tls-certs\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.677418 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.677398    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-654ht\" (UniqueName: \"kubernetes.io/projected/3c97b261-f060-45a3-95e5-a6f924b51047-kube-api-access-654ht\") pod \"auth-test-model-0a962a-kserve-7f75465669-rs2ng\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") " pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:32.778025 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:32.777995    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:33.466467 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:33.466420    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"]
Oct 30 18:59:33.674281 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:33.666240    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"]
Oct 30 18:59:33.674281 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:33.666520    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc" podUID="2dd5599e-0c92-4050-9351-6eef1270a0ae" containerName="maas-api" containerID="cri-o://9c7a1bcb86d1ebe3f507ab80d006266af775eedfa6f434ec4d233671be28ceb6" gracePeriod=30
Oct 30 18:59:34.368060 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.368002    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f308cee-8550-48a4-b417-0f49cde47af8" path="/var/lib/kubelet/pods/6f308cee-8550-48a4-b417-0f49cde47af8/volumes"
Oct 30 18:59:34.383561 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.381514    4372 generic.go:356] "Generic (PLEG): container finished" podID="2dd5599e-0c92-4050-9351-6eef1270a0ae" containerID="9c7a1bcb86d1ebe3f507ab80d006266af775eedfa6f434ec4d233671be28ceb6" exitCode=0
Oct 30 18:59:34.385436 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.385413    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d81ca16-a936-4c2b-adee-174cdc1e3266" path="/var/lib/kubelet/pods/8d81ca16-a936-4c2b-adee-174cdc1e3266/volumes"
Oct 30 18:59:34.386097 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.386074    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng" event={"ID":"3c97b261-f060-45a3-95e5-a6f924b51047","Type":"ContainerStarted","Data":"40b1737878a157569517bc689ba364bf965a9b09aa31eb7faa06159fd73792bf"}
Oct 30 18:59:34.391293 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.391271    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng" event={"ID":"3c97b261-f060-45a3-95e5-a6f924b51047","Type":"ContainerStarted","Data":"f47cf9dbcad7f9270ff336abee97040b6bfd6a102cacf7dd1eca979901714321"}
Oct 30 18:59:34.391451 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.391436    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc" event={"ID":"2dd5599e-0c92-4050-9351-6eef1270a0ae","Type":"ContainerDied","Data":"9c7a1bcb86d1ebe3f507ab80d006266af775eedfa6f434ec4d233671be28ceb6"}
Oct 30 18:59:34.793168 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.778186    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:34.847439 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.847411    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:59:34.876579 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.876301    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng" podStartSLOduration=2.876286306 podStartE2EDuration="2.876286306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:34.406339569 +0000 UTC m=+1588.274757900" watchObservedRunningTime="2026-10-30 18:59:34.876286306 +0000 UTC m=+1588.744704635"
Oct 30 18:59:34.946767 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.946730    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd5599e-0c92-4050-9351-6eef1270a0ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd5599e-0c92-4050-9351-6eef1270a0ae-openshift-service-ca\") pod \"2dd5599e-0c92-4050-9351-6eef1270a0ae\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") "
Oct 30 18:59:34.946992 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.946927    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-maas-api-tls\") pod \"2dd5599e-0c92-4050-9351-6eef1270a0ae\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") "
Oct 30 18:59:34.947201 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.947172    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd5599e-0c92-4050-9351-6eef1270a0ae-openshift-service-ca" pod "2dd5599e-0c92-4050-9351-6eef1270a0ae" (UID: "2dd5599e-0c92-4050-9351-6eef1270a0ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:34.947201 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.947191    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-metrics-certs\") pod \"2dd5599e-0c92-4050-9351-6eef1270a0ae\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") "
Oct 30 18:59:34.947350 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.947312    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd5599e-0c92-4050-9351-6eef1270a0ae-kube-api-access-vcxzh\" (UniqueName: \"kubernetes.io/projected/2dd5599e-0c92-4050-9351-6eef1270a0ae-kube-api-access-vcxzh\") pod \"2dd5599e-0c92-4050-9351-6eef1270a0ae\" (UID: \"2dd5599e-0c92-4050-9351-6eef1270a0ae\") "
Oct 30 18:59:34.947934 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.947898    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd5599e-0c92-4050-9351-6eef1270a0ae-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:34.949565 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.949538    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd5599e-0c92-4050-9351-6eef1270a0ae-kube-api-access-vcxzh" pod "2dd5599e-0c92-4050-9351-6eef1270a0ae" (UID: "2dd5599e-0c92-4050-9351-6eef1270a0ae"). InnerVolumeSpecName "kube-api-access-vcxzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:34.949660 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.949562    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-metrics-certs" pod "2dd5599e-0c92-4050-9351-6eef1270a0ae" (UID: "2dd5599e-0c92-4050-9351-6eef1270a0ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:34.950189 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:34.950172    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-maas-api-tls" pod "2dd5599e-0c92-4050-9351-6eef1270a0ae" (UID: "2dd5599e-0c92-4050-9351-6eef1270a0ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:35.049548 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:35.049460    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:35.049548 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:35.049504    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd5599e-0c92-4050-9351-6eef1270a0ae-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:35.049548 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:35.049521    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcxzh\" (UniqueName: \"kubernetes.io/projected/2dd5599e-0c92-4050-9351-6eef1270a0ae-kube-api-access-vcxzh\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:35.427753 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:35.427720    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"
Oct 30 18:59:35.429306 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:35.429277    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc" event={"ID":"2dd5599e-0c92-4050-9351-6eef1270a0ae","Type":"ContainerDied","Data":"c7afcb8a6a213687ccfcd0703536dbfdec8e979387628b7a0b5566df2b99aa2d"}
Oct 30 18:59:35.429488 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:35.429474    4372 scope.go:122] "RemoveContainer" containerID="9c7a1bcb86d1ebe3f507ab80d006266af775eedfa6f434ec4d233671be28ceb6"
Oct 30 18:59:35.561747 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:35.560214    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"]
Oct 30 18:59:35.652185 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:35.638000    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bcb5af-7cb98886df-nchsc"]
Oct 30 18:59:36.277277 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:36.277244    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd5599e-0c92-4050-9351-6eef1270a0ae" path="/var/lib/kubelet/pods/2dd5599e-0c92-4050-9351-6eef1270a0ae/volumes"
Oct 30 18:59:40.434702 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.434661    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"]
Oct 30 18:59:40.435322 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.435306    4372 cpu_manager.go:414] "containerMap: removing container" podUID="2dd5599e-0c92-4050-9351-6eef1270a0ae" containerName="maas-api"
Oct 30 18:59:40.435375 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.435323    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd5599e-0c92-4050-9351-6eef1270a0ae" containerName="maas-api"
Oct 30 18:59:40.435447 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.435438    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd5599e-0c92-4050-9351-6eef1270a0ae" containerName="maas-api"
Oct 30 18:59:40.440603 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.440579    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"
Oct 30 18:59:40.452753 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.452731    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-6d7870\""
Oct 30 18:59:40.452936 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.452902    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-6d7870-dockercfg-rtskz\""
Oct 30 18:59:40.454916 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.454892    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"]
Oct 30 18:59:40.504156 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.493968    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"]
Oct 30 18:59:40.509434 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.509398    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"
Oct 30 18:59:40.545447 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.545406    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp22k\" (UniqueName: \"kubernetes.io/projected/31d23f77-b321-4ed8-b1e2-e0d1376d5697-kube-api-access-wp22k\") pod \"payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj\" (UID: \"31d23f77-b321-4ed8-b1e2-e0d1376d5697\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"
Oct 30 18:59:40.546193 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.546161    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d23f77-b321-4ed8-b1e2-e0d1376d5697-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj\" (UID: \"31d23f77-b321-4ed8-b1e2-e0d1376d5697\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"
Oct 30 18:59:40.546590 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.546564    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9c8g\" (UniqueName: \"kubernetes.io/projected/925f7b63-9cb5-4ee8-8867-91530abaf46c-kube-api-access-t9c8g\") pod \"payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg\" (UID: \"925f7b63-9cb5-4ee8-8867-91530abaf46c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"
Oct 30 18:59:40.546760 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.546707    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/925f7b63-9cb5-4ee8-8867-91530abaf46c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg\" (UID: \"925f7b63-9cb5-4ee8-8867-91530abaf46c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"
Oct 30 18:59:40.549626 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.549579    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"]
Oct 30 18:59:40.679151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.678041    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9c8g\" (UniqueName: \"kubernetes.io/projected/925f7b63-9cb5-4ee8-8867-91530abaf46c-kube-api-access-t9c8g\") pod \"payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg\" (UID: \"925f7b63-9cb5-4ee8-8867-91530abaf46c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"
Oct 30 18:59:40.679151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.678200    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/925f7b63-9cb5-4ee8-8867-91530abaf46c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg\" (UID: \"925f7b63-9cb5-4ee8-8867-91530abaf46c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"
Oct 30 18:59:40.679151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.678309    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp22k\" (UniqueName: \"kubernetes.io/projected/31d23f77-b321-4ed8-b1e2-e0d1376d5697-kube-api-access-wp22k\") pod \"payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj\" (UID: \"31d23f77-b321-4ed8-b1e2-e0d1376d5697\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"
Oct 30 18:59:40.679151 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.678374    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d23f77-b321-4ed8-b1e2-e0d1376d5697-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj\" (UID: \"31d23f77-b321-4ed8-b1e2-e0d1376d5697\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"
Oct 30 18:59:40.679511 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.679477    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d23f77-b321-4ed8-b1e2-e0d1376d5697-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj\" (UID: \"31d23f77-b321-4ed8-b1e2-e0d1376d5697\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"
Oct 30 18:59:40.734215 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.680611    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/925f7b63-9cb5-4ee8-8867-91530abaf46c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg\" (UID: \"925f7b63-9cb5-4ee8-8867-91530abaf46c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"
Oct 30 18:59:40.792164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.783360    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp22k\" (UniqueName: \"kubernetes.io/projected/31d23f77-b321-4ed8-b1e2-e0d1376d5697-kube-api-access-wp22k\") pod \"payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj\" (UID: \"31d23f77-b321-4ed8-b1e2-e0d1376d5697\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"
Oct 30 18:59:40.822182 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.814964    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9c8g\" (UniqueName: \"kubernetes.io/projected/925f7b63-9cb5-4ee8-8867-91530abaf46c-kube-api-access-t9c8g\") pod \"payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg\" (UID: \"925f7b63-9cb5-4ee8-8867-91530abaf46c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"
Oct 30 18:59:40.840208 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:40.839685    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"
Oct 30 18:59:41.058233 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:41.058191    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"
Oct 30 18:59:41.258694 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:41.258667    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"]
Oct 30 18:59:41.259663 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:59:41.259636    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d23f77_b321_4ed8_b1e2_e0d1376d5697.slice/crio-fb87e4773f09879e11f18385a187ad614ac4fca144a9e786e9ca4574d4631586 WatchSource:0}: Error finding container fb87e4773f09879e11f18385a187ad614ac4fca144a9e786e9ca4574d4631586: Status 404 returned error can't find the container with id fb87e4773f09879e11f18385a187ad614ac4fca144a9e786e9ca4574d4631586
Oct 30 18:59:41.399345 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:41.399317    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"]
Oct 30 18:59:41.399451 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:59:41.399339    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925f7b63_9cb5_4ee8_8867_91530abaf46c.slice/crio-16c5a1ab4478ae742975b28dfd5ee702dbd68e314ee3c7ea5d7980e28f13a785 WatchSource:0}: Error finding container 16c5a1ab4478ae742975b28dfd5ee702dbd68e314ee3c7ea5d7980e28f13a785: Status 404 returned error can't find the container with id 16c5a1ab4478ae742975b28dfd5ee702dbd68e314ee3c7ea5d7980e28f13a785
Oct 30 18:59:41.501000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:41.500966    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj" event={"ID":"31d23f77-b321-4ed8-b1e2-e0d1376d5697","Type":"ContainerStarted","Data":"f39689a3d7ca0e8f1404c4dbddaf919c832160c01853e7f69cf93c0b8491bb08"}
Oct 30 18:59:41.501528 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:41.501027    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj" event={"ID":"31d23f77-b321-4ed8-b1e2-e0d1376d5697","Type":"ContainerStarted","Data":"fb87e4773f09879e11f18385a187ad614ac4fca144a9e786e9ca4574d4631586"}
Oct 30 18:59:41.501528 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:41.501185    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"
Oct 30 18:59:41.502724 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:41.502690    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg" event={"ID":"925f7b63-9cb5-4ee8-8867-91530abaf46c","Type":"ContainerStarted","Data":"6a23df6fdc6c45da962a2f48b6ea1c831a4bc3eab426affc3766c7a98e26e256"}
Oct 30 18:59:41.502861 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:41.502749    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg" event={"ID":"925f7b63-9cb5-4ee8-8867-91530abaf46c","Type":"ContainerStarted","Data":"16c5a1ab4478ae742975b28dfd5ee702dbd68e314ee3c7ea5d7980e28f13a785"}
Oct 30 18:59:41.503096 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:41.503076    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"
Oct 30 18:59:41.547207 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:41.546798    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj" podStartSLOduration=1.5467758630000001 podStartE2EDuration="1.546775863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:41.527793141 +0000 UTC m=+1595.396211485" watchObservedRunningTime="2026-10-30 18:59:41.546775863 +0000 UTC m=+1595.415194194"
Oct 30 18:59:42.513862 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:42.513775    4372 generic.go:356] "Generic (PLEG): container finished" podID="3028a9d2-a829-4420-a287-ce7c2ea64334" containerID="4935fa0b08a5dfb1063a20173ba3824513c0377ebd70401a42d9a3368f6013a8" exitCode=0
Oct 30 18:59:42.513862 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:42.513846    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6" event={"ID":"3028a9d2-a829-4420-a287-ce7c2ea64334","Type":"ContainerDied","Data":"4935fa0b08a5dfb1063a20173ba3824513c0377ebd70401a42d9a3368f6013a8"}
Oct 30 18:59:42.514443 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:42.514425    4372 scope.go:122] "RemoveContainer" containerID="4935fa0b08a5dfb1063a20173ba3824513c0377ebd70401a42d9a3368f6013a8"
Oct 30 18:59:42.516257 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:42.516222    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:59:42.516348 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:42.516315    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:59:42.516388 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:42.516365    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:59:42.544442 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:42.544347    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg" podStartSLOduration=2.544328252 podStartE2EDuration="2.544328252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:41.638514999 +0000 UTC m=+1595.506933330" watchObservedRunningTime="2026-10-30 18:59:42.544328252 +0000 UTC m=+1596.412746582"
Oct 30 18:59:42.780169 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:42.779143    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:42.794607 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:42.794553    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:43.522524 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:43.522492    4372 generic.go:356] "Generic (PLEG): container finished" podID="13a85f8c-9244-499b-a12f-37d11fb59cba" containerID="048efa7ee353ef701682ffe878dd6679f7618964afceb5ae301a663401b692f4" exitCode=0
Oct 30 18:59:43.522980 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:43.522564    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv" event={"ID":"13a85f8c-9244-499b-a12f-37d11fb59cba","Type":"ContainerDied","Data":"048efa7ee353ef701682ffe878dd6679f7618964afceb5ae301a663401b692f4"}
Oct 30 18:59:43.523062 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:43.523046    4372 scope.go:122] "RemoveContainer" containerID="048efa7ee353ef701682ffe878dd6679f7618964afceb5ae301a663401b692f4"
Oct 30 18:59:43.524930 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:43.524906    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6" event={"ID":"3028a9d2-a829-4420-a287-ce7c2ea64334","Type":"ContainerStarted","Data":"3cf3d14557f167f0ad1afd6d9ea195cd58cea5f19ada9078b6959c9214bdf7cb"}
Oct 30 18:59:43.525056 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:43.524901    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:59:43.525056 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:43.525009    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:59:43.525056 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:43.525048    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:59:43.537218 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:43.537200    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 18:59:44.262183 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:44.262142    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 18:59:44.262358 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:44.262220    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6" podUID="3028a9d2-a829-4420-a287-ce7c2ea64334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 18:59:44.269158 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:44.269116    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:59:44.392164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:44.386233    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:59:44.539076 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:44.538973    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv" event={"ID":"13a85f8c-9244-499b-a12f-37d11fb59cba","Type":"ContainerStarted","Data":"297cc4497c14c70280ad7a9a78c604284c1811b68a13cdd5f93b4fefbcfabc08"}
Oct 30 18:59:45.270150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:45.265819    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 18:59:45.270150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:45.266042    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6" podUID="3028a9d2-a829-4420-a287-ce7c2ea64334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 18:59:45.471663 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:45.471631    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:59:45.472355 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:45.472333    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 18:59:45.472409 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:45.472392    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv" podUID="13a85f8c-9244-499b-a12f-37d11fb59cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 18:59:46.260472 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:46.260440    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 18:59:46.260888 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:46.260496    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6" podUID="3028a9d2-a829-4420-a287-ce7c2ea64334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 18:59:46.471716 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:46.471682    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 18:59:46.471883 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:46.471734    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv" podUID="13a85f8c-9244-499b-a12f-37d11fb59cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 18:59:47.136150 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:47.129303    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:59:47.265064 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:47.264662    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 18:59:47.265064 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:47.264713    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6" podUID="3028a9d2-a829-4420-a287-ce7c2ea64334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 18:59:47.471731 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:47.471654    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 18:59:47.471731 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:47.471709    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv" podUID="13a85f8c-9244-499b-a12f-37d11fb59cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 18:59:48.259971 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:48.259934    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 18:59:48.260164 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:48.259991    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6" podUID="3028a9d2-a829-4420-a287-ce7c2ea64334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 18:59:48.472190 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:48.472121    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 18:59:48.472574 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:48.472219    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv" podUID="13a85f8c-9244-499b-a12f-37d11fb59cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 18:59:49.264695 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:49.264657    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 18:59:49.264998 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:49.264964    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6" podUID="3028a9d2-a829-4420-a287-ce7c2ea64334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 18:59:49.472088 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:49.471954    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 18:59:49.472088 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:49.472007    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv" podUID="13a85f8c-9244-499b-a12f-37d11fb59cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 18:59:50.274042 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:50.274006    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:59:50.275105 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:50.275079    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 18:59:50.549119 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:50.549034    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:59:50.549854 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:50.549832    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:59:50.550506 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:50.550481    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 18:59:50.857802 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:50.857769    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"]
Oct 30 18:59:50.862253 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:50.862232    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:50.886782 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:50.886752    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"]
Oct 30 18:59:50.887245 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:50.887222    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-6d7870-serving-cert\""
Oct 30 18:59:50.926461 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:50.926431    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db876\" (UniqueName: \"kubernetes.io/projected/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-kube-api-access-db876\") pod \"maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:50.926688 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:50.926667    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:50.926841 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:50.926813    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:50.926895 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:50.926874    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:51.028298 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:51.028272    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:51.028411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:51.028311    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:51.028411 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:51.028365    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db876\" (UniqueName: \"kubernetes.io/projected/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-kube-api-access-db876\") pod \"maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:51.028544 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:51.028450    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:51.029000 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:51.028977    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:51.030966 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:51.030946    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:51.031045 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:51.031020    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:51.072806 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:51.072778    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db876\" (UniqueName: \"kubernetes.io/projected/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-kube-api-access-db876\") pod \"maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:51.187399 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:51.184217    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:51.497981 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:51.497952    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"]
Oct 30 18:59:51.498653 ip-10-0-142-153 kubenswrapper[4372]: W1030 18:59:51.498628    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfcf9c2_e42d_443c_ae5c_d734dbd4bff7.slice/crio-ae0746a5527c3a956273d910708a4e90a68bd5c298ec1f5dd62d2c6e642b9aa0 WatchSource:0}: Error finding container ae0746a5527c3a956273d910708a4e90a68bd5c298ec1f5dd62d2c6e642b9aa0: Status 404 returned error can't find the container with id ae0746a5527c3a956273d910708a4e90a68bd5c298ec1f5dd62d2c6e642b9aa0
Oct 30 18:59:51.679370 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:51.679328    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn" event={"ID":"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7","Type":"ContainerStarted","Data":"ae0746a5527c3a956273d910708a4e90a68bd5c298ec1f5dd62d2c6e642b9aa0"}
Oct 30 18:59:52.517039 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:52.517012    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"
Oct 30 18:59:52.517230 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:52.517076    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"
Oct 30 18:59:52.687279 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:52.687240    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn" event={"ID":"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7","Type":"ContainerStarted","Data":"faf45de0c81f6c4f1261e3b6a5421f841ade7efa28ebe0d2615564c4522121cc"}
Oct 30 18:59:52.687730 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:52.687324    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:52.710698 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:52.710656    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn" podStartSLOduration=2.151284428 podStartE2EDuration="2.71064527s" totalImagesPullingTime="559.360842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:52.710436249 +0000 UTC m=+1606.578854580" watchObservedRunningTime="2026-10-30 18:59:52.71064527 +0000 UTC m=+1606.579063600"
Oct 30 18:59:56.643137 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:56.643092    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 18:59:57.312580 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:57.312548    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"]
Oct 30 18:59:57.316851 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:57.316829    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"
Oct 30 18:59:57.344560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:57.330978    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-784fbf-dockercfg-jpxl5\""
Oct 30 18:59:57.344560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:57.334978    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-784fbf\""
Oct 30 18:59:57.344560 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:57.339997    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"]
Oct 30 18:59:57.446524 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:57.446489    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef41158-9775-4f62-a109-49e3d21649c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg\" (UID: \"3ef41158-9775-4f62-a109-49e3d21649c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"
Oct 30 18:59:57.446696 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:57.446621    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnl44\" (UniqueName: \"kubernetes.io/projected/3ef41158-9775-4f62-a109-49e3d21649c9-kube-api-access-lnl44\") pod \"payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg\" (UID: \"3ef41158-9775-4f62-a109-49e3d21649c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"
Oct 30 18:59:57.550754 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:57.549036    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnl44\" (UniqueName: \"kubernetes.io/projected/3ef41158-9775-4f62-a109-49e3d21649c9-kube-api-access-lnl44\") pod \"payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg\" (UID: \"3ef41158-9775-4f62-a109-49e3d21649c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"
Oct 30 18:59:57.550754 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:57.549341    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef41158-9775-4f62-a109-49e3d21649c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg\" (UID: \"3ef41158-9775-4f62-a109-49e3d21649c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"
Oct 30 18:59:57.550754 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:57.550422    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef41158-9775-4f62-a109-49e3d21649c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg\" (UID: \"3ef41158-9775-4f62-a109-49e3d21649c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"
Oct 30 18:59:57.626900 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:57.626855    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnl44\" (UniqueName: \"kubernetes.io/projected/3ef41158-9775-4f62-a109-49e3d21649c9-kube-api-access-lnl44\") pod \"payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg\" (UID: \"3ef41158-9775-4f62-a109-49e3d21649c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"
Oct 30 18:59:57.648465 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:57.648425    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"
Oct 30 18:59:57.766709 ip-10-0-142-153 kubenswrapper[4372]: E1030 18:59:57.766649    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 18:59:58.088853 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:58.088798    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"]
Oct 30 18:59:58.725345 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:58.725315    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 18:59:58.819928 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:58.819886    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg" event={"ID":"3ef41158-9775-4f62-a109-49e3d21649c9","Type":"ContainerStarted","Data":"3b31a0f19ba40a68471d2e737335a1703eaa055581388ec3365da50893fb1728"}
Oct 30 18:59:58.820108 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:58.819958    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg" event={"ID":"3ef41158-9775-4f62-a109-49e3d21649c9","Type":"ContainerStarted","Data":"67bb8993a023c6b51a7dc04dfc6875a7b5e6525c599f5e6b940e678e0b9ccda0"}
Oct 30 18:59:58.820723 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:58.820699    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"
Oct 30 18:59:58.849683 ip-10-0-142-153 kubenswrapper[4372]: I1030 18:59:58.849625    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg" podStartSLOduration=1.849609517 podStartE2EDuration="1.849609517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:58.846352002 +0000 UTC m=+1612.714770332" watchObservedRunningTime="2026-10-30 18:59:58.849609517 +0000 UTC m=+1612.718027848"
Oct 30 19:00:00.270322 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:00.270294    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"]
Oct 30 19:00:00.274745 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:00.274728    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"
Oct 30 19:00:00.317417 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:00.317389    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"]
Oct 30 19:00:00.363423 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:00.363398    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:00:00.367257 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:00.367231    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chhht\" (UniqueName: \"kubernetes.io/projected/d70ff4e7-fab4-4b15-94f3-f2a1abf30e42-kube-api-access-chhht\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44\" (UID: \"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"
Oct 30 19:00:00.367431 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:00.367412    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d70ff4e7-fab4-4b15-94f3-f2a1abf30e42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44\" (UID: \"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"
Oct 30 19:00:00.468633 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:00.468603    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d70ff4e7-fab4-4b15-94f3-f2a1abf30e42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44\" (UID: \"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"
Oct 30 19:00:00.468771 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:00.468755    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chhht\" (UniqueName: \"kubernetes.io/projected/d70ff4e7-fab4-4b15-94f3-f2a1abf30e42-kube-api-access-chhht\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44\" (UID: \"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"
Oct 30 19:00:00.469315 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:00.469292    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d70ff4e7-fab4-4b15-94f3-f2a1abf30e42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44\" (UID: \"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"
Oct 30 19:00:00.504402 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:00.504367    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chhht\" (UniqueName: \"kubernetes.io/projected/d70ff4e7-fab4-4b15-94f3-f2a1abf30e42-kube-api-access-chhht\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44\" (UID: \"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"
Oct 30 19:00:00.626919 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:00.626890    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"
Oct 30 19:00:00.942300 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:00.942270    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"]
Oct 30 19:00:00.942417 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:00:00.942298    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70ff4e7_fab4_4b15_94f3_f2a1abf30e42.slice/crio-a259997640f6d42afc3a9f8234e04ba2303650d49ae4ec77156423386c88aa60 WatchSource:0}: Error finding container a259997640f6d42afc3a9f8234e04ba2303650d49ae4ec77156423386c88aa60: Status 404 returned error can't find the container with id a259997640f6d42afc3a9f8234e04ba2303650d49ae4ec77156423386c88aa60
Oct 30 19:00:01.842135 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:01.842101    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44" event={"ID":"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42","Type":"ContainerStarted","Data":"4da68b808eb23c75dc174bde1c6e48cf90797e83cda757db10b1043bb0ba8af8"}
Oct 30 19:00:01.842536 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:01.842201    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44" event={"ID":"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42","Type":"ContainerStarted","Data":"a259997640f6d42afc3a9f8234e04ba2303650d49ae4ec77156423386c88aa60"}
Oct 30 19:00:01.892154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:01.873097    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44" podStartSLOduration=1.873076219 podStartE2EDuration="1.873076219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:01.870366135 +0000 UTC m=+1615.738784465" watchObservedRunningTime="2026-10-30 19:00:01.873076219 +0000 UTC m=+1615.741494549"
Oct 30 19:00:02.850475 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:02.850439    4372 generic.go:356] "Generic (PLEG): container finished" podID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42" containerID="4da68b808eb23c75dc174bde1c6e48cf90797e83cda757db10b1043bb0ba8af8" exitCode=60
Oct 30 19:00:02.850859 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:02.850502    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44" event={"ID":"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42","Type":"ContainerDied","Data":"4da68b808eb23c75dc174bde1c6e48cf90797e83cda757db10b1043bb0ba8af8"}
Oct 30 19:00:02.850859 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:02.850789    4372 scope.go:122] "RemoveContainer" containerID="4da68b808eb23c75dc174bde1c6e48cf90797e83cda757db10b1043bb0ba8af8"
Oct 30 19:00:03.873928 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:03.873857    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44" event={"ID":"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42","Type":"ContainerStarted","Data":"240d194cf8ee4af81e34fa94964711f3d4b550b30cf2fc93feb2041aaf765c3d"}
Oct 30 19:00:04.929039 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:04.929000    4372 generic.go:356] "Generic (PLEG): container finished" podID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42" containerID="240d194cf8ee4af81e34fa94964711f3d4b550b30cf2fc93feb2041aaf765c3d" exitCode=60
Oct 30 19:00:04.929509 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:04.929340    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44" event={"ID":"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42","Type":"ContainerDied","Data":"240d194cf8ee4af81e34fa94964711f3d4b550b30cf2fc93feb2041aaf765c3d"}
Oct 30 19:00:04.929509 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:04.929406    4372 scope.go:122] "RemoveContainer" containerID="4da68b808eb23c75dc174bde1c6e48cf90797e83cda757db10b1043bb0ba8af8"
Oct 30 19:00:04.929939 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:04.929919    4372 scope.go:122] "RemoveContainer" containerID="240d194cf8ee4af81e34fa94964711f3d4b550b30cf2fc93feb2041aaf765c3d"
Oct 30 19:00:04.930404 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:00:04.930375    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44_odh-ai-gateway-infra(d70ff4e7-fab4-4b15-94f3-f2a1abf30e42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44" podUID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42"
Oct 30 19:00:05.942941 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:05.942911    4372 scope.go:122] "RemoveContainer" containerID="240d194cf8ee4af81e34fa94964711f3d4b550b30cf2fc93feb2041aaf765c3d"
Oct 30 19:00:05.943438 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:00:05.943354    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44_odh-ai-gateway-infra(d70ff4e7-fab4-4b15-94f3-f2a1abf30e42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44" podUID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42"
Oct 30 19:00:08.128561 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.128525    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"]
Oct 30 19:00:08.135979 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.135956    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.142637 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.142607    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"]
Oct 30 19:00:08.153616 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.153589    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ab7df0\"/\"openshift-service-ca.crt\""
Oct 30 19:00:08.153750 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.153676    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ab7df0\"/\"kube-root-ca.crt\""
Oct 30 19:00:08.154841 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.154814    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ab7df0\"/\"default-dockercfg-w8lk8\""
Oct 30 19:00:08.154841 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.154835    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ab7df0\"/\"auth-test-model-ab7df0-kserve-self-signed-certs\""
Oct 30 19:00:08.180922 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.180888    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-tmp-dir\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.181077 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.181056    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-home\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.181202 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.181164    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-model-cache\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.181345 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.181299    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngkf7\" (UniqueName: \"kubernetes.io/projected/4549f795-bff1-43c4-8f1a-5493b5f975d8-kube-api-access-ngkf7\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.181412 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.181386    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4549f795-bff1-43c4-8f1a-5493b5f975d8-tls-certs\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.181476 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.181447    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-dshm\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.282077 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.282052    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkf7\" (UniqueName: \"kubernetes.io/projected/4549f795-bff1-43c4-8f1a-5493b5f975d8-kube-api-access-ngkf7\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.286049 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.282104    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4549f795-bff1-43c4-8f1a-5493b5f975d8-tls-certs\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.286049 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.282171    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-dshm\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.286049 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.282194    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-tmp-dir\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.286049 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.282308    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-home\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.286049 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.282350    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-model-cache\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.286049 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.282689    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-home\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.286049 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.282803    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-model-cache\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.286049 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.282868    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-tmp-dir\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.286049 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.284578    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-dshm\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.286049 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.284992    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4549f795-bff1-43c4-8f1a-5493b5f975d8-tls-certs\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.294634 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.294611    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngkf7\" (UniqueName: \"kubernetes.io/projected/4549f795-bff1-43c4-8f1a-5493b5f975d8-kube-api-access-ngkf7\") pod \"auth-test-model-ab7df0-kserve-89fbfff7-w2gwk\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") " pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.457204 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.456875    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:08.788420 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.788396    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"]
Oct 30 19:00:08.788815 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:00:08.788790    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4549f795_bff1_43c4_8f1a_5493b5f975d8.slice/crio-f3e059408c0445a2c98db46f11947cd32957ad0d35951ba6a5b85adb83853772 WatchSource:0}: Error finding container f3e059408c0445a2c98db46f11947cd32957ad0d35951ba6a5b85adb83853772: Status 404 returned error can't find the container with id f3e059408c0445a2c98db46f11947cd32957ad0d35951ba6a5b85adb83853772
Oct 30 19:00:08.978470 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.978438    4372 generic.go:356] "Generic (PLEG): container finished" podID="36220b42-54dc-4dd2-aac8-e44b93f61a55" containerID="dccce248504137e4bd9e5c375b4d844071c7f1692e23f9eb75e1c57e7b1b4ecb" exitCode=0
Oct 30 19:00:08.978648 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.978505    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd" event={"ID":"36220b42-54dc-4dd2-aac8-e44b93f61a55","Type":"ContainerDied","Data":"dccce248504137e4bd9e5c375b4d844071c7f1692e23f9eb75e1c57e7b1b4ecb"}
Oct 30 19:00:08.978975 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.978953    4372 scope.go:122] "RemoveContainer" containerID="dccce248504137e4bd9e5c375b4d844071c7f1692e23f9eb75e1c57e7b1b4ecb"
Oct 30 19:00:08.980395 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.980362    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk" event={"ID":"4549f795-bff1-43c4-8f1a-5493b5f975d8","Type":"ContainerStarted","Data":"51a93e2a5e29772a0eba742cbd35f1c134529a073225b19dd580d990a5383841"}
Oct 30 19:00:08.980483 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.980396    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk" event={"ID":"4549f795-bff1-43c4-8f1a-5493b5f975d8","Type":"ContainerStarted","Data":"f3e059408c0445a2c98db46f11947cd32957ad0d35951ba6a5b85adb83853772"}
Oct 30 19:00:08.980683 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.980656    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:08.980737 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.980725    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:08.980777 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:08.980760    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:09.032931 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:09.032873    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk" podStartSLOduration=1.03285466 podStartE2EDuration="1.03285466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:09.027650556 +0000 UTC m=+1622.896068887" watchObservedRunningTime="2026-10-30 19:00:09.03285466 +0000 UTC m=+1622.901272992"
Oct 30 19:00:09.458290 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:09.458252    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:09.831739 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:09.831711    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"
Oct 30 19:00:10.016093 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:10.016050    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd" event={"ID":"36220b42-54dc-4dd2-aac8-e44b93f61a55","Type":"ContainerStarted","Data":"bf4cf844c1d99bc99a86d755f0a3397cf9f583670311c9771aaba696c841e138"}
Oct 30 19:00:10.544462 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:10.544307    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:00:10.544462 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:10.544387    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd" podUID="36220b42-54dc-4dd2-aac8-e44b93f61a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 19:00:10.545036 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:10.544686    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 19:00:11.543777 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:11.543741    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:00:11.543966 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:11.543792    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd" podUID="36220b42-54dc-4dd2-aac8-e44b93f61a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 19:00:12.543878 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.543844    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:00:12.544317 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.543894    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd" podUID="36220b42-54dc-4dd2-aac8-e44b93f61a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 19:00:12.719581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.719544    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"]
Oct 30 19:00:12.724510 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.724486    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.735478 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.735444    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-model-cache\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.735626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.735536    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-dshm\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.735626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.735752    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-tmp-dir\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.735626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.735788    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-home\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.735626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.735819    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp9vc\" (UniqueName: \"kubernetes.io/projected/4fcba8ac-337e-4ee9-9828-a9869f075500-kube-api-access-kp9vc\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.735626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.735908    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcba8ac-337e-4ee9-9828-a9869f075500-tls-certs\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.735626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.742097    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-304f22\"/\"openshift-service-ca.crt\""
Oct 30 19:00:12.735626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.742210    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-304f22\"/\"kube-root-ca.crt\""
Oct 30 19:00:12.735626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.742413    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-304f22\"/\"default-dockercfg-5sfnd\""
Oct 30 19:00:12.735626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.742539    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-304f22\"/\"auth-test-model-304f22-kserve-self-signed-certs\""
Oct 30 19:00:12.755402 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.755373    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"]
Oct 30 19:00:12.836935 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.836901    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-model-cache\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.837096 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.836952    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-dshm\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.837181 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.837117    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-tmp-dir\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.837181 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.837173    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-home\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.837276 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.837200    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9vc\" (UniqueName: \"kubernetes.io/projected/4fcba8ac-337e-4ee9-9828-a9869f075500-kube-api-access-kp9vc\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.837324 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.837291    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcba8ac-337e-4ee9-9828-a9869f075500-tls-certs\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.837398 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.837370    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-model-cache\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.837489 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.837448    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-tmp-dir\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.837552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.837534    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-home\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.839422 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.839395    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-dshm\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.839779 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.839762    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcba8ac-337e-4ee9-9828-a9869f075500-tls-certs\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:12.850004 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:12.849985    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp9vc\" (UniqueName: \"kubernetes.io/projected/4fcba8ac-337e-4ee9-9828-a9869f075500-kube-api-access-kp9vc\") pod \"auth-test-model-304f22-kserve-658758d7fc-gdhrw\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") " pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:13.069149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:13.056180    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:13.410469 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:00:13.410437    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fcba8ac_337e_4ee9_9828_a9869f075500.slice/crio-8457c22d26402c4c2b151f91fd34eaac73a872ff1e25f9d5828dfad1dbd7fc4d WatchSource:0}: Error finding container 8457c22d26402c4c2b151f91fd34eaac73a872ff1e25f9d5828dfad1dbd7fc4d: Status 404 returned error can't find the container with id 8457c22d26402c4c2b151f91fd34eaac73a872ff1e25f9d5828dfad1dbd7fc4d
Oct 30 19:00:13.412069 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:13.412042    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"]
Oct 30 19:00:13.543515 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:13.543486    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:00:13.543696 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:13.543535    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd" podUID="36220b42-54dc-4dd2-aac8-e44b93f61a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 19:00:14.110731 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:14.110685    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw" event={"ID":"4fcba8ac-337e-4ee9-9828-a9869f075500","Type":"ContainerStarted","Data":"90bd040460fcc342365cc1523899bca09ad9b581541e8b905d728549c03f3f99"}
Oct 30 19:00:14.110731 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:14.110729    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw" event={"ID":"4fcba8ac-337e-4ee9-9828-a9869f075500","Type":"ContainerStarted","Data":"8457c22d26402c4c2b151f91fd34eaac73a872ff1e25f9d5828dfad1dbd7fc4d"}
Oct 30 19:00:14.135824 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:14.135774    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw" podStartSLOduration=2.135759571 podStartE2EDuration="2.135759571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:14.131968789 +0000 UTC m=+1628.000387123" watchObservedRunningTime="2026-10-30 19:00:14.135759571 +0000 UTC m=+1628.004177933"
Oct 30 19:00:14.547204 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:14.545310    4372 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:00:14.547204 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:14.545376    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd" podUID="36220b42-54dc-4dd2-aac8-e44b93f61a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 19:00:15.061246 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:15.061212    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:15.257945 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:15.257910    4372 scope.go:122] "RemoveContainer" containerID="240d194cf8ee4af81e34fa94964711f3d4b550b30cf2fc93feb2041aaf765c3d"
Oct 30 19:00:15.548162 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:15.548103    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 19:00:15.548439 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:15.548411    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 19:00:15.549224 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:15.549200    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 19:00:16.128684 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:16.128659    4372 generic.go:356] "Generic (PLEG): container finished" podID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42" containerID="98a2c6ad57d50d55948bfe839ea69506993c2c76fd963ed8393a8adcfee28cb2" exitCode=60
Oct 30 19:00:16.128796 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:16.128727    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44" event={"ID":"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42","Type":"ContainerDied","Data":"98a2c6ad57d50d55948bfe839ea69506993c2c76fd963ed8393a8adcfee28cb2"}
Oct 30 19:00:16.128796 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:16.128784    4372 scope.go:122] "RemoveContainer" containerID="240d194cf8ee4af81e34fa94964711f3d4b550b30cf2fc93feb2041aaf765c3d"
Oct 30 19:00:16.129274 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:16.129252    4372 scope.go:122] "RemoveContainer" containerID="98a2c6ad57d50d55948bfe839ea69506993c2c76fd963ed8393a8adcfee28cb2"
Oct 30 19:00:16.129587 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:00:16.129565    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44_odh-ai-gateway-infra(d70ff4e7-fab4-4b15-94f3-f2a1abf30e42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44" podUID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42"
Oct 30 19:00:17.174860 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:17.174831    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"]
Oct 30 19:00:17.349758 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:17.349735    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"
Oct 30 19:00:17.399887 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:17.399844    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70ff4e7-fab4-4b15-94f3-f2a1abf30e42-kube-api-access-chhht\" (UniqueName: \"kubernetes.io/projected/d70ff4e7-fab4-4b15-94f3-f2a1abf30e42-kube-api-access-chhht\") pod \"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42\" (UID: \"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42\") "
Oct 30 19:00:17.400044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:17.400027    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70ff4e7-fab4-4b15-94f3-f2a1abf30e42-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d70ff4e7-fab4-4b15-94f3-f2a1abf30e42-maas-api-service-ca\") pod \"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42\" (UID: \"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42\") "
Oct 30 19:00:17.400394 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:17.400365    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70ff4e7-fab4-4b15-94f3-f2a1abf30e42-maas-api-service-ca" pod "d70ff4e7-fab4-4b15-94f3-f2a1abf30e42" (UID: "d70ff4e7-fab4-4b15-94f3-f2a1abf30e42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:17.400650 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:17.400632    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d70ff4e7-fab4-4b15-94f3-f2a1abf30e42-maas-api-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:17.402677 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:17.402654    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70ff4e7-fab4-4b15-94f3-f2a1abf30e42-kube-api-access-chhht" pod "d70ff4e7-fab4-4b15-94f3-f2a1abf30e42" (UID: "d70ff4e7-fab4-4b15-94f3-f2a1abf30e42"). InnerVolumeSpecName "kube-api-access-chhht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:17.502351 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:17.502294    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chhht\" (UniqueName: \"kubernetes.io/projected/d70ff4e7-fab4-4b15-94f3-f2a1abf30e42-kube-api-access-chhht\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.162355 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:18.162323    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"
Oct 30 19:00:18.162555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:18.162368    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44" event={"ID":"d70ff4e7-fab4-4b15-94f3-f2a1abf30e42","Type":"ContainerDied","Data":"a259997640f6d42afc3a9f8234e04ba2303650d49ae4ec77156423386c88aa60"}
Oct 30 19:00:18.162555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:18.162466    4372 scope.go:122] "RemoveContainer" containerID="98a2c6ad57d50d55948bfe839ea69506993c2c76fd963ed8393a8adcfee28cb2"
Oct 30 19:00:18.195496 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:18.195466    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"]
Oct 30 19:00:18.211313 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:18.211286    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889780-lft44"]
Oct 30 19:00:18.332859 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:18.332820    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42" path="/var/lib/kubelet/pods/d70ff4e7-fab4-4b15-94f3-f2a1abf30e42/volumes"
Oct 30 19:00:18.458330 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:18.458255    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:18.476509 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:18.476486    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:19.195186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:19.195157    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:00:23.061594 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:23.061558    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:23.073764 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:23.073741    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:23.270488 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:23.270457    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:00:27.697094 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:00:27.697052    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:00:36.765097 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.765066    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"]
Oct 30 19:00:36.765732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.765715    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42" containerName="cleanup"
Oct 30 19:00:36.765732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.765733    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42" containerName="cleanup"
Oct 30 19:00:36.765836 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.765744    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42" containerName="cleanup"
Oct 30 19:00:36.765836 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.765750    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42" containerName="cleanup"
Oct 30 19:00:36.765939 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.765910    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42" containerName="cleanup"
Oct 30 19:00:36.765939 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.765919    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42" containerName="cleanup"
Oct 30 19:00:36.770223 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.770200    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.777447 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.777424    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-784fbf-openshift-default-dockercfg-xjmtj\""
Oct 30 19:00:36.791384 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.791358    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"]
Oct 30 19:00:36.881684 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.880249    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-workload-certs\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.881684 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.880352    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-workload-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.881684 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.880387    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-podinfo\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.881684 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.880535    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r25tq\" (UniqueName: \"kubernetes.io/projected/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-kube-api-access-r25tq\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.881684 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.880605    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-token\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.881684 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.880662    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-data\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.881684 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.880698    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-envoy\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.881684 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.880722    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istiod-ca-cert\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.881684 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.880820    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-credential-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.982250 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.982216    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-token\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.982451 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.982276    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-data\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.982451 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.982315    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-envoy\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.982451 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.982340    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istiod-ca-cert\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.982649 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.982557    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-credential-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.982714 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.982664    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-workload-certs\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.982776 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.982754    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-workload-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.982843 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.982790    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-podinfo\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.982929 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.982904    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-credential-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.982995 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.982784    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-data\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.982995 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.982974    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r25tq\" (UniqueName: \"kubernetes.io/projected/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-kube-api-access-r25tq\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.983104 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.983062    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-workload-certs\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.983104 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.983067    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-workload-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.983378 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.983357    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istiod-ca-cert\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.985256 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.985236    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-envoy\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.985466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.985449    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-podinfo\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:36.999217 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:36.999188    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r25tq\" (UniqueName: \"kubernetes.io/projected/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-kube-api-access-r25tq\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:37.004430 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:37.004409    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-token\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:37.087509 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:37.087482    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:37.381944 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:00:37.381912    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d1c3fc_26d4_40ea_ab5a_9d2758bedbd4.slice/crio-0b37fc37cc8f90e1022b78aa7d3f6414d48394e9f4b0370e1e5178d471e24e91 WatchSource:0}: Error finding container 0b37fc37cc8f90e1022b78aa7d3f6414d48394e9f4b0370e1e5178d471e24e91: Status 404 returned error can't find the container with id 0b37fc37cc8f90e1022b78aa7d3f6414d48394e9f4b0370e1e5178d471e24e91
Oct 30 19:00:37.382441 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:37.382416    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"]
Oct 30 19:00:37.385059 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:37.385027    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:37.385190 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:37.385105    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:37.385190 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:37.385155    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:38.410785 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:38.410737    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb" event={"ID":"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4","Type":"ContainerStarted","Data":"52ecf7652f868d7d9cde9f9a039d49ed1c9f562cf2395ac24c3b45a7c09cb153"}
Oct 30 19:00:38.411239 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:38.410798    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb" event={"ID":"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4","Type":"ContainerStarted","Data":"0b37fc37cc8f90e1022b78aa7d3f6414d48394e9f4b0370e1e5178d471e24e91"}
Oct 30 19:00:38.452001 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:38.451950    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb" podStartSLOduration=2.451937749 podStartE2EDuration="2.451937749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:38.451664685 +0000 UTC m=+1652.320083014" watchObservedRunningTime="2026-10-30 19:00:38.451937749 +0000 UTC m=+1652.320356078"
Oct 30 19:00:39.101627 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:39.091478    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:39.108153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:39.103738    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 19:00:39.108153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:39.103816    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb" podUID="b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 19:00:40.094159 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:40.093261    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 19:00:40.094159 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:40.093320    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb" podUID="b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 19:00:41.100152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:41.099486    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 19:00:41.100152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:41.099545    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb" podUID="b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 19:00:42.089481 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:42.089443    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 19:00:42.090888 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:42.090851    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb" podUID="b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 19:00:43.097154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:43.093427    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 19:00:43.097154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:43.093515    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb" podUID="b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 19:00:44.104483 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:44.104442    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:44.106964 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:44.106928    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:44.113497 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:44.113454    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:00:45.381152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.378013    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"]
Oct 30 19:00:45.381152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.379152    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42" containerName="cleanup"
Oct 30 19:00:45.381152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.379186    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42" containerName="cleanup"
Oct 30 19:00:45.381152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.379402    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70ff4e7-fab4-4b15-94f3-f2a1abf30e42" containerName="cleanup"
Oct 30 19:00:45.385229 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.385202    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.403925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.403895    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-6d7870\"/\"openshift-service-ca.crt\""
Oct 30 19:00:45.404227 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.403989    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-6d7870\"/\"kube-root-ca.crt\""
Oct 30 19:00:45.404227 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.404053    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-6d7870\"/\"default-dockercfg-94fnp\""
Oct 30 19:00:45.404227 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.404102    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-6d7870\"/\"auth-test-model-6d7870-kserve-self-signed-certs\""
Oct 30 19:00:45.407423 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.407379    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"]
Oct 30 19:00:45.470402 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.470366    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-model-cache\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.470539 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.470518    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-home\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.470595 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.470544    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-dshm\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.470595 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.470575    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1875d62-38cc-452b-87f0-5b78523a0d41-tls-certs\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.470742 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.470718    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-tmp-dir\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.470789 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.470776    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnvjr\" (UniqueName: \"kubernetes.io/projected/e1875d62-38cc-452b-87f0-5b78523a0d41-kube-api-access-nnvjr\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.588184 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.582185    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-home\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.614184 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.613235    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-dshm\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.614184 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.613301    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1875d62-38cc-452b-87f0-5b78523a0d41-tls-certs\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.614184 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.613427    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-tmp-dir\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.614184 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.613528    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnvjr\" (UniqueName: \"kubernetes.io/projected/e1875d62-38cc-452b-87f0-5b78523a0d41-kube-api-access-nnvjr\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.614184 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.613814    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-model-cache\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.614603 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.614395    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-model-cache\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.614603 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.589631    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-home\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.650271 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.638474    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-tmp-dir\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.767015 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.766982    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1875d62-38cc-452b-87f0-5b78523a0d41-tls-certs\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.767292 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.767033    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-dshm\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:45.773632 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:45.773606    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnvjr\" (UniqueName: \"kubernetes.io/projected/e1875d62-38cc-452b-87f0-5b78523a0d41-kube-api-access-nnvjr\") pod \"auth-test-model-6d7870-kserve-54c956f97-wsrbt\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") " pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:46.007876 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.007799    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:46.275803 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.275731    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"]
Oct 30 19:00:46.281592 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.281563    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.285774 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.285752    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-6d7870-openshift-default-dockercfg-sb8km\""
Oct 30 19:00:46.301010 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.300987    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"]
Oct 30 19:00:46.309960 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.309926    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"]
Oct 30 19:00:46.370690 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.370661    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-token\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.370805 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.370708    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-workload-socket\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.370805 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.370737    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-data\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.370917 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.370872    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-workload-certs\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.370976 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.370915    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istiod-ca-cert\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.371058 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.371039    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq6nj\" (UniqueName: \"kubernetes.io/projected/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-kube-api-access-nq6nj\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.371115 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.371102    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-podinfo\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.371265 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.371250    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-envoy\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.371360 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.371346    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-credential-socket\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.473243 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.473212    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-token\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.473811 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.473254    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-workload-socket\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.473811 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.473281    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-data\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.473811 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.473431    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-workload-certs\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.473811 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.473480    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istiod-ca-cert\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.473811 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.473546    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq6nj\" (UniqueName: \"kubernetes.io/projected/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-kube-api-access-nq6nj\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.473811 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.473684    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-podinfo\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.473811 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.473762    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-envoy\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.473811 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.473818    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-data\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.474634 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.473870    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-workload-certs\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.474634 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.473890    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-credential-socket\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.474634 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.473907    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-workload-socket\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.474634 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.474221    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-credential-socket\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.474634 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.474304    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istiod-ca-cert\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.475518 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.475489    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt" event={"ID":"e1875d62-38cc-452b-87f0-5b78523a0d41","Type":"ContainerStarted","Data":"3a2349e82c8b8653bb59ebf0e8ba370e551991bec3b567ac922f547610ff635d"}
Oct 30 19:00:46.475629 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.475529    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt" event={"ID":"e1875d62-38cc-452b-87f0-5b78523a0d41","Type":"ContainerStarted","Data":"cb0863c6487939bc8ad168d03a3870d7991c4b22d9c86ef5295ddd97c0a0b340"}
Oct 30 19:00:46.476328 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.476310    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-envoy\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.476473 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.476456    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-podinfo\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.485842 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.485823    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq6nj\" (UniqueName: \"kubernetes.io/projected/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-kube-api-access-nq6nj\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.486155 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.486119    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-token\") pod \"e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") " pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.499362 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.499322    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt" podStartSLOduration=1.499311567 podStartE2EDuration="1.499311567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:46.496322066 +0000 UTC m=+1660.364740397" watchObservedRunningTime="2026-10-30 19:00:46.499311567 +0000 UTC m=+1660.367729929"
Oct 30 19:00:46.634882 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.634560    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:46.982618 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.982589    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"]
Oct 30 19:00:46.983031 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:00:46.983005    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ac9c50_e716_4e3e_9b75_befcdb4e4dbb.slice/crio-cc64fb73e29610b5c34afcb4d782792833c10d47228033a2aaef6c7b4cccade4 WatchSource:0}: Error finding container cc64fb73e29610b5c34afcb4d782792833c10d47228033a2aaef6c7b4cccade4: Status 404 returned error can't find the container with id cc64fb73e29610b5c34afcb4d782792833c10d47228033a2aaef6c7b4cccade4
Oct 30 19:00:46.985615 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.985584    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:46.985680 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.985648    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:46.985718 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:46.985678    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:47.008179 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:47.008159    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:47.485711 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:47.485667    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7" event={"ID":"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb","Type":"ContainerStarted","Data":"5c71ceabb5b00a02480c2ff052829c36145a2cfc7267ec4b77b1e23ca8e5d882"}
Oct 30 19:00:47.486151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:47.485737    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7" event={"ID":"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb","Type":"ContainerStarted","Data":"cc64fb73e29610b5c34afcb4d782792833c10d47228033a2aaef6c7b4cccade4"}
Oct 30 19:00:47.518318 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:47.518250    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7" podStartSLOduration=1.5182314620000001 podStartE2EDuration="1.518231462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:47.511599454 +0000 UTC m=+1661.380017807" watchObservedRunningTime="2026-10-30 19:00:47.518231462 +0000 UTC m=+1661.386649792"
Oct 30 19:00:47.636081 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:47.635965    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:48.637611 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:48.637571    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 19:00:48.638019 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:48.637624    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7" podUID="74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 19:00:49.147279 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.143109    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"]
Oct 30 19:00:49.224423 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.224375    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.225625 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.225524    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"]
Oct 30 19:00:49.245061 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.245035    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-743e0b-openshift-default-dockercfg-dntsq\""
Oct 30 19:00:49.303983 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.303951    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-workload-certs\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.304178 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.304000    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-envoy\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.304178 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.304088    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-podinfo\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.304281 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.304213    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q6cg\" (UniqueName: \"kubernetes.io/projected/c2964aa0-b86e-47bc-ab9c-8f277d20e691-kube-api-access-9q6cg\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.304360 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.304293    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-token\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.304416 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.304402    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-credential-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.304450 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.304427    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-workload-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.304593 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.304573    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-data\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.304643 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.304625    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istiod-ca-cert\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.406088 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.406003    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-credential-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.406088 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.406045    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-workload-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.406361 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.406192    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-data\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.406361 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.406235    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istiod-ca-cert\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.406361 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.406336    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-workload-certs\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.406523 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.406391    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-envoy\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.406523 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.406421    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-podinfo\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.406523 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.406470    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q6cg\" (UniqueName: \"kubernetes.io/projected/c2964aa0-b86e-47bc-ab9c-8f277d20e691-kube-api-access-9q6cg\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.406523 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.406483    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-credential-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.406741 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.406564    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-token\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.407075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.407049    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-workload-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.407259 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.407236    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-workload-certs\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.407538 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.407517    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-data\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.408080 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.408004    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istiod-ca-cert\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.409752 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.409725    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-envoy\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.409930 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.409913    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-podinfo\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.423683 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.423658    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-token\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.423809 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.423792    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q6cg\" (UniqueName: \"kubernetes.io/projected/c2964aa0-b86e-47bc-ab9c-8f277d20e691-kube-api-access-9q6cg\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.542020 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.541991    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:49.636248 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.635735    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 19:00:49.636248 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.635779    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7" podUID="74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 19:00:49.853096 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:00:49.853058    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2964aa0_b86e_47bc_ab9c_8f277d20e691.slice/crio-8283cb74e71bdfdd3f1fad871d113b3652f217a09b52ba16793ce08efb8e698c WatchSource:0}: Error finding container 8283cb74e71bdfdd3f1fad871d113b3652f217a09b52ba16793ce08efb8e698c: Status 404 returned error can't find the container with id 8283cb74e71bdfdd3f1fad871d113b3652f217a09b52ba16793ce08efb8e698c
Oct 30 19:00:49.853481 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.853459    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"]
Oct 30 19:00:49.855907 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.855873    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:49.855982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.855943    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:49.855982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:49.855974    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:50.541152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:50.533838    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t" event={"ID":"c2964aa0-b86e-47bc-ab9c-8f277d20e691","Type":"ContainerStarted","Data":"b1688a99011f4d4242c7376c3e21d2e8b0392f55b31f5d7ee922512346b05c4e"}
Oct 30 19:00:50.541152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:50.533903    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t" event={"ID":"c2964aa0-b86e-47bc-ab9c-8f277d20e691","Type":"ContainerStarted","Data":"8283cb74e71bdfdd3f1fad871d113b3652f217a09b52ba16793ce08efb8e698c"}
Oct 30 19:00:50.543201 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:50.542316    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:50.543973 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:50.543946    4372 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 19:00:50.544101 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:50.543986    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t" podUID="c2964aa0-b86e-47bc-ab9c-8f277d20e691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 19:00:50.567391 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:50.567349    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t" podStartSLOduration=1.567337365 podStartE2EDuration="1.567337365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:50.561618152 +0000 UTC m=+1664.430036483" watchObservedRunningTime="2026-10-30 19:00:50.567337365 +0000 UTC m=+1664.435755695"
Oct 30 19:00:50.636081 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:50.636050    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 19:00:50.636266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:50.636095    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7" podUID="74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 19:00:51.542359 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.542323    4372 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 19:00:51.542794 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.542375    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t" podUID="c2964aa0-b86e-47bc-ab9c-8f277d20e691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 19:00:51.635775 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.635735    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 19:00:51.636010 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.635797    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7" podUID="74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 19:00:51.856493 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.856332    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"]
Oct 30 19:00:51.861961 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.861920    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"]
Oct 30 19:00:51.862154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.862027    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:51.956934 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.956210    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-workload-certs\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:51.956934 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.956262    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044b9f16-2b46-4246-b997-7791ba4fc3e8-istiod-ca-cert\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:51.956934 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.956356    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-envoy\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:51.956934 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.956434    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-credential-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:51.956934 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.956489    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-podinfo\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:51.956934 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.956543    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tct5\" (UniqueName: \"kubernetes.io/projected/044b9f16-2b46-4246-b997-7791ba4fc3e8-kube-api-access-6tct5\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:51.956934 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.956584    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-token\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:51.956934 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.956613    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-data\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:51.956934 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:51.956751    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-workload-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.058409 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.058371    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-credential-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.058583 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.058431    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-podinfo\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.058583 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.058467    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tct5\" (UniqueName: \"kubernetes.io/projected/044b9f16-2b46-4246-b997-7791ba4fc3e8-kube-api-access-6tct5\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.058583 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.058493    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-token\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.058583 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.058514    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-data\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.058718 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.058644    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-workload-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.058902 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.058834    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-credential-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.058902 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.058833    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-workload-certs\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.058902 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.058902    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044b9f16-2b46-4246-b997-7791ba4fc3e8-istiod-ca-cert\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.059197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.058929    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-envoy\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.059197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.059036    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-data\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.059315 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.059226    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-workload-certs\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.059315 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.059253    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-workload-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.059571 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.059553    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044b9f16-2b46-4246-b997-7791ba4fc3e8-istiod-ca-cert\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.061304 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.061283    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-envoy\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.062427 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.062403    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-podinfo\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.072811 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.072783    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-token\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.073311 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.073278    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tct5\" (UniqueName: \"kubernetes.io/projected/044b9f16-2b46-4246-b997-7791ba4fc3e8-kube-api-access-6tct5\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.179721 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.178471    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:52.543432 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.543397    4372 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 19:00:52.543943 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.543456    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t" podUID="c2964aa0-b86e-47bc-ab9c-8f277d20e691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 19:00:52.635749 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.635601    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 19:00:52.635749 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.635663    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7" podUID="74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 19:00:52.655833 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.655808    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"]
Oct 30 19:00:52.656717 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:00:52.656696    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044b9f16_2b46_4246_b997_7791ba4fc3e8.slice/crio-4778cea7c92018ff32b1fbd86ff1108f3864e00217f9fd054a947637e6931136 WatchSource:0}: Error finding container 4778cea7c92018ff32b1fbd86ff1108f3864e00217f9fd054a947637e6931136: Status 404 returned error can't find the container with id 4778cea7c92018ff32b1fbd86ff1108f3864e00217f9fd054a947637e6931136
Oct 30 19:00:52.659698 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.659666    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:52.659775 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.659731    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:52.659775 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:52.659761    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:00:53.543235 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:53.543199    4372 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 19:00:53.543419 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:53.543252    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t" podUID="c2964aa0-b86e-47bc-ab9c-8f277d20e691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 19:00:53.560300 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:53.560264    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85" event={"ID":"044b9f16-2b46-4246-b997-7791ba4fc3e8","Type":"ContainerStarted","Data":"30877e67666e50f0d78374b3757ddaca7a454ed45c39f659b3d3f3516e6a5c4c"}
Oct 30 19:00:53.560709 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:53.560330    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85" event={"ID":"044b9f16-2b46-4246-b997-7791ba4fc3e8","Type":"ContainerStarted","Data":"4778cea7c92018ff32b1fbd86ff1108f3864e00217f9fd054a947637e6931136"}
Oct 30 19:00:53.610387 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:53.610334    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85" podStartSLOduration=2.610319206 podStartE2EDuration="2.610319206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:53.605580112 +0000 UTC m=+1667.473998444" watchObservedRunningTime="2026-10-30 19:00:53.610319206 +0000 UTC m=+1667.478737601"
Oct 30 19:00:53.640827 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:53.640799    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:53.641250 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:53.641228    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:53.641888 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:53.641870    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:00:54.200151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:54.199450    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:00:54.204616 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:54.204580    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 19:00:54.204769 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:54.204658    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85" podUID="044b9f16-2b46-4246-b997-7791ba4fc3e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 19:00:54.543118 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:54.543039    4372 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 19:00:54.543118 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:54.543085    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t" podUID="c2964aa0-b86e-47bc-ab9c-8f277d20e691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 19:00:55.192151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:55.186263    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 19:00:55.192151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:55.186353    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85" podUID="044b9f16-2b46-4246-b997-7791ba4fc3e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 19:00:55.542822 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:55.542727    4372 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 19:00:55.542822 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:55.542782    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t" podUID="c2964aa0-b86e-47bc-ab9c-8f277d20e691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 19:00:56.010314 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:56.010280    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:56.038633 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:56.038601    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:56.190158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:56.185359    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 19:00:56.190158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:56.185416    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85" podUID="044b9f16-2b46-4246-b997-7791ba4fc3e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 19:00:56.623567 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:56.623535    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:00:56.828345 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:56.828306    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:56.829213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:56.829190    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:56.829826 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:56.829806    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:00:57.203500 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:57.203463    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 19:00:57.203703 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:57.203527    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85" podUID="044b9f16-2b46-4246-b997-7791ba4fc3e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 19:00:57.714366 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:00:57.714325    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:00:58.180797 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:58.180754    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 19:00:58.181079 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:58.181049    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85" podUID="044b9f16-2b46-4246-b997-7791ba4fc3e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 19:00:59.191472 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:59.191435    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 19:00:59.192051 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:00:59.192021    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85" podUID="044b9f16-2b46-4246-b997-7791ba4fc3e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 19:01:00.180092 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:00.180058    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 19:01:00.180308 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:00.180117    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85" podUID="044b9f16-2b46-4246-b997-7791ba4fc3e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 19:01:01.185215 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:01.185182    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:01:01.185680 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:01.185646    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:01:01.186710 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:01.186686    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:01:04.004069 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:04.004034    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:01:16.598172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:16.598139    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"]
Oct 30 19:01:16.598835 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:16.598792    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t" podUID="c2964aa0-b86e-47bc-ab9c-8f277d20e691" containerName="istio-proxy" containerID="cri-o://b1688a99011f4d4242c7376c3e21d2e8b0392f55b31f5d7ee922512346b05c4e" gracePeriod=30
Oct 30 19:01:21.931057 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:21.930980    4372 generic.go:356] "Generic (PLEG): container finished" podID="c2964aa0-b86e-47bc-ab9c-8f277d20e691" containerID="b1688a99011f4d4242c7376c3e21d2e8b0392f55b31f5d7ee922512346b05c4e" exitCode=0
Oct 30 19:01:21.931057 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:21.931042    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t" event={"ID":"c2964aa0-b86e-47bc-ab9c-8f277d20e691","Type":"ContainerDied","Data":"b1688a99011f4d4242c7376c3e21d2e8b0392f55b31f5d7ee922512346b05c4e"}
Oct 30 19:01:22.502262 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.500730    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:01:22.614558 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.614511    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istiod-ca-cert\") pod \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") "
Oct 30 19:01:22.615440 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.615415    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-token\" (UniqueName: \"kubernetes.io/projected/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-token\") pod \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") "
Oct 30 19:01:22.615665 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.615643    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-podinfo\") pod \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") "
Oct 30 19:01:22.615931 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.615910    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-workload-socket\") pod \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") "
Oct 30 19:01:22.616091 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.616073    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-credential-socket\") pod \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") "
Oct 30 19:01:22.616333 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.616315    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-data\") pod \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") "
Oct 30 19:01:22.616792 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.616772    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-envoy\") pod \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") "
Oct 30 19:01:22.616916 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.616903    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2964aa0-b86e-47bc-ab9c-8f277d20e691-kube-api-access-9q6cg\" (UniqueName: \"kubernetes.io/projected/c2964aa0-b86e-47bc-ab9c-8f277d20e691-kube-api-access-9q6cg\") pod \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") "
Oct 30 19:01:22.617054 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.617039    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-workload-certs\") pod \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\" (UID: \"c2964aa0-b86e-47bc-ab9c-8f277d20e691\") "
Oct 30 19:01:22.620688 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.615316    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istiod-ca-cert" pod "c2964aa0-b86e-47bc-ab9c-8f277d20e691" (UID: "c2964aa0-b86e-47bc-ab9c-8f277d20e691"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:22.620688 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.620282    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-credential-socket" pod "c2964aa0-b86e-47bc-ab9c-8f277d20e691" (UID: "c2964aa0-b86e-47bc-ab9c-8f277d20e691"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:22.620688 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.620602    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-data" pod "c2964aa0-b86e-47bc-ab9c-8f277d20e691" (UID: "c2964aa0-b86e-47bc-ab9c-8f277d20e691"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:22.621217 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.621194    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-workload-socket" pod "c2964aa0-b86e-47bc-ab9c-8f277d20e691" (UID: "c2964aa0-b86e-47bc-ab9c-8f277d20e691"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:22.625079 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.625056    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-workload-certs" pod "c2964aa0-b86e-47bc-ab9c-8f277d20e691" (UID: "c2964aa0-b86e-47bc-ab9c-8f277d20e691"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:22.643376 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.643328    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-token" pod "c2964aa0-b86e-47bc-ab9c-8f277d20e691" (UID: "c2964aa0-b86e-47bc-ab9c-8f277d20e691"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:22.643687 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.643659    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-envoy" pod "c2964aa0-b86e-47bc-ab9c-8f277d20e691" (UID: "c2964aa0-b86e-47bc-ab9c-8f277d20e691"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:22.645376 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.645338    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2964aa0-b86e-47bc-ab9c-8f277d20e691-kube-api-access-9q6cg" pod "c2964aa0-b86e-47bc-ab9c-8f277d20e691" (UID: "c2964aa0-b86e-47bc-ab9c-8f277d20e691"). InnerVolumeSpecName "kube-api-access-9q6cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:22.647290 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.647260    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-podinfo" pod "c2964aa0-b86e-47bc-ab9c-8f277d20e691" (UID: "c2964aa0-b86e-47bc-ab9c-8f277d20e691"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:01:22.720652 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.720628    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.720750 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.720657    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.720750 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.720672    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.720750 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.720685    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.720750 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.720702    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.720750 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.720716    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.720750 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.720730    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.720750 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.720746    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2964aa0-b86e-47bc-ab9c-8f277d20e691-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.720980 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.720760    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q6cg\" (UniqueName: \"kubernetes.io/projected/c2964aa0-b86e-47bc-ab9c-8f277d20e691-kube-api-access-9q6cg\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.939791 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.939757    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t" event={"ID":"c2964aa0-b86e-47bc-ab9c-8f277d20e691","Type":"ContainerDied","Data":"8283cb74e71bdfdd3f1fad871d113b3652f217a09b52ba16793ce08efb8e698c"}
Oct 30 19:01:22.940179 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.939817    4372 scope.go:122] "RemoveContainer" containerID="b1688a99011f4d4242c7376c3e21d2e8b0392f55b31f5d7ee922512346b05c4e"
Oct 30 19:01:22.940179 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.939768    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"
Oct 30 19:01:22.975694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.975668    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"]
Oct 30 19:01:22.982662 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:22.982639    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-65l6t"]
Oct 30 19:01:24.266181 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:24.266153    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2964aa0-b86e-47bc-ab9c-8f277d20e691" path="/var/lib/kubelet/pods/c2964aa0-b86e-47bc-ab9c-8f277d20e691/volumes"
Oct 30 19:01:27.696585 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:01:27.696547    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:01:29.851173 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.851137    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"]
Oct 30 19:01:29.852876 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.852849    4372 cpu_manager.go:414] "containerMap: removing container" podUID="c2964aa0-b86e-47bc-ab9c-8f277d20e691" containerName="istio-proxy"
Oct 30 19:01:29.852876 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.852876    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2964aa0-b86e-47bc-ab9c-8f277d20e691" containerName="istio-proxy"
Oct 30 19:01:29.853103 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.853083    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2964aa0-b86e-47bc-ab9c-8f277d20e691" containerName="istio-proxy"
Oct 30 19:01:29.857314 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.857292    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:29.866535 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.866513    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"]
Oct 30 19:01:29.869824 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.869799    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-03362d-openshift-default-dockercfg-rgbtn\""
Oct 30 19:01:29.946770 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.946735    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-workload-certs\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:29.946952 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.946776    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-token\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:29.946952 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.946926    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-podinfo\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:29.947067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.946975    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-workload-socket\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:29.947172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.947156    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istiod-ca-cert\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:29.947224 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.947193    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-data\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:29.947285 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.947267    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-credential-socket\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:29.947383 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.947369    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck8hr\" (UniqueName: \"kubernetes.io/projected/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-kube-api-access-ck8hr\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:29.947424 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:29.947403    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-envoy\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.048373 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.048346    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-podinfo\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.048565 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.048409    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-workload-socket\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.048565 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.048463    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istiod-ca-cert\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.048565 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.048488    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-data\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.048733 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.048568    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-credential-socket\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.048733 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.048630    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck8hr\" (UniqueName: \"kubernetes.io/projected/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-kube-api-access-ck8hr\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.048733 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.048658    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-envoy\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.048899 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.048860    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-workload-socket\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.048968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.048907    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-workload-certs\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.048968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.048915    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-data\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.048968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.048944    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-token\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.049169 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.049091    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-credential-socket\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.049378 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.049245    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-workload-certs\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.049378 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.049323    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istiod-ca-cert\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.051016 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.050996    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-envoy\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.051285 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.051269    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-podinfo\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.066270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.066244    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-token\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.067243 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.067220    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck8hr\" (UniqueName: \"kubernetes.io/projected/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-kube-api-access-ck8hr\") pod \"e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") " pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.174092 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.174037    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:30.467525 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.467497    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"]
Oct 30 19:01:30.467684 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:01:30.467662    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb01e2fe_8077_4f8b_98e2_b107af18e2d4.slice/crio-f5ce1651f942913f03523aa78b2e9905707cf02cf1eeafbe1f983b3a57913b86 WatchSource:0}: Error finding container f5ce1651f942913f03523aa78b2e9905707cf02cf1eeafbe1f983b3a57913b86: Status 404 returned error can't find the container with id f5ce1651f942913f03523aa78b2e9905707cf02cf1eeafbe1f983b3a57913b86
Oct 30 19:01:30.470200 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.470169    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:01:30.470278 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.470239    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:01:30.470278 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:30.470268    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:01:31.016740 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:31.016699    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6" event={"ID":"eb01e2fe-8077-4f8b-98e2-b107af18e2d4","Type":"ContainerStarted","Data":"9fa075200e13e549bb073ba5cc6b16e5cb8d08cd4fb71ae02d9620f433fbf500"}
Oct 30 19:01:31.017188 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:31.016796    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6" event={"ID":"eb01e2fe-8077-4f8b-98e2-b107af18e2d4","Type":"ContainerStarted","Data":"f5ce1651f942913f03523aa78b2e9905707cf02cf1eeafbe1f983b3a57913b86"}
Oct 30 19:01:31.061938 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:31.061881    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6" podStartSLOduration=2.061866635 podStartE2EDuration="2.061866635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:31.058352755 +0000 UTC m=+1704.926771086" watchObservedRunningTime="2026-10-30 19:01:31.061866635 +0000 UTC m=+1704.930284964"
Oct 30 19:01:31.174700 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:31.174672    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:31.175874 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:31.175850    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 19:01:31.176005 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:31.175893    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6" podUID="eb01e2fe-8077-4f8b-98e2-b107af18e2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 19:01:32.174763 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:32.174729    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 19:01:32.175221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:32.174798    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6" podUID="eb01e2fe-8077-4f8b-98e2-b107af18e2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 19:01:33.175017 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:33.174979    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 19:01:33.175510 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:33.175046    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6" podUID="eb01e2fe-8077-4f8b-98e2-b107af18e2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 19:01:34.238151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:34.181232    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 19:01:34.238798 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:34.238760    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6" podUID="eb01e2fe-8077-4f8b-98e2-b107af18e2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 19:01:35.175213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:35.175183    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 19:01:35.175392 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:35.175248    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6" podUID="eb01e2fe-8077-4f8b-98e2-b107af18e2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 19:01:36.175033 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:36.174995    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 19:01:36.175457 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:36.175071    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6" podUID="eb01e2fe-8077-4f8b-98e2-b107af18e2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 19:01:37.175578 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:37.175525    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 19:01:37.176240 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:37.176204    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6" podUID="eb01e2fe-8077-4f8b-98e2-b107af18e2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 19:01:38.187389 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:38.187350    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:38.188063 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:38.188023    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:38.188915 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:01:38.188893    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:01:57.695874 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:01:57.695837    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:02:05.673020 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:02:05.672984    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:02:27.697249 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:02:27.697214    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:02:48.774501 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:02:48.774463    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:02:57.760514 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:02:57.760478    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:03:08.877456 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:03:08.877424    4372 scope.go:122] "RemoveContainer" containerID="ef37a6325d5e2820bab4b28935f17235178346e364ef6ad808a70e2296dcb057"
Oct 30 19:03:27.700005 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:03:27.699968    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:03:36.861019 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:03:36.860985    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:03:57.700207 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:03:57.700085    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:04:02.847872 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:02.847842    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"]
Oct 30 19:04:02.852269 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:02.852250    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:02.864599 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:02.864571    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a717c7-serving-cert\""
Oct 30 19:04:02.872066 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:02.870436    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"]
Oct 30 19:04:03.026148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:03.022807    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:03.026148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:03.022973    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-metrics-certs\") pod \"maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:03.026148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:03.023049    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j2pv\" (UniqueName: \"kubernetes.io/projected/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-kube-api-access-6j2pv\") pod \"maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:03.045297 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:03.045256    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:03.147377 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:03.147298    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:03.147532 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:03.147424    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-metrics-certs\") pod \"maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:03.147532 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:03.147486    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j2pv\" (UniqueName: \"kubernetes.io/projected/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-kube-api-access-6j2pv\") pod \"maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:03.147532 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:03.147512    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:03.148144 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:03.148098    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:03.150106 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:03.150085    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-metrics-certs\") pod \"maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:03.150286 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:03.150265    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:03.161760 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:03.161736    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j2pv\" (UniqueName: \"kubernetes.io/projected/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-kube-api-access-6j2pv\") pod \"maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:03.174534 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:03.174511    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:03.504780 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:04:03.504749    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae7c57e_6d9f_4fa1_a9cc_0256ba10fab7.slice/crio-119e52266dff1fc273be3a3444b9b89c7180bac1e4e013f05c382d8b713e2af1 WatchSource:0}: Error finding container 119e52266dff1fc273be3a3444b9b89c7180bac1e4e013f05c382d8b713e2af1: Status 404 returned error can't find the container with id 119e52266dff1fc273be3a3444b9b89c7180bac1e4e013f05c382d8b713e2af1
Oct 30 19:04:03.505074 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:03.505048    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"]
Oct 30 19:04:04.231213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:04.229111    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb" event={"ID":"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7","Type":"ContainerStarted","Data":"91a9b747fd6f3f405b76c5bd822673a87f182b4dc158b41a867da6b7937e8718"}
Oct 30 19:04:04.231213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:04.229363    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:04.231213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:04.229393    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb" event={"ID":"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7","Type":"ContainerStarted","Data":"119e52266dff1fc273be3a3444b9b89c7180bac1e4e013f05c382d8b713e2af1"}
Oct 30 19:04:04.254069 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:04.254022    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb" podStartSLOduration=1.733495784 podStartE2EDuration="2.254009331s" totalImagesPullingTime="520.513547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:04.251025376 +0000 UTC m=+1858.119443729" watchObservedRunningTime="2026-10-30 19:04:04.254009331 +0000 UTC m=+1858.122427712"
Oct 30 19:04:05.340739 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:05.340700    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"]
Oct 30 19:04:09.757943 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:09.757907    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"]
Oct 30 19:04:10.243415 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:10.242514    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:04:12.470616 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:12.470580    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:04:14.321158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:14.310950    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"]
Oct 30 19:04:14.321158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:14.311616    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" podUID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" containerName="istio-proxy" containerID="cri-o://6f0a9e869f2f9e60e06c8c2e6e5ee23a60ae7299fafbc645bb68f042cbc3d4fd" gracePeriod=30
Oct 30 19:04:16.680262 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:16.680215    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7408-0caf3b"]
Oct 30 19:04:16.689150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:16.687481    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7408-0caf3b"
Oct 30 19:04:16.697694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:16.694350    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7h66b\""
Oct 30 19:04:16.697694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:16.695385    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 19:04:16.697694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:16.695472    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 19:04:16.703147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:16.701186    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7408-0caf3b"]
Oct 30 19:04:16.716432 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:16.715843    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275k2\" (UniqueName: \"kubernetes.io/projected/8e1ec39f-1c6a-4442-b4da-6f1383a9db2d-kube-api-access-275k2\") pod \"test-curl-7408-0caf3b\" (UID: \"8e1ec39f-1c6a-4442-b4da-6f1383a9db2d\") " pod="opendatahub/test-curl-7408-0caf3b"
Oct 30 19:04:16.818043 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:16.818007    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275k2\" (UniqueName: \"kubernetes.io/projected/8e1ec39f-1c6a-4442-b4da-6f1383a9db2d-kube-api-access-275k2\") pod \"test-curl-7408-0caf3b\" (UID: \"8e1ec39f-1c6a-4442-b4da-6f1383a9db2d\") " pod="opendatahub/test-curl-7408-0caf3b"
Oct 30 19:04:16.832721 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:16.832694    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-275k2\" (UniqueName: \"kubernetes.io/projected/8e1ec39f-1c6a-4442-b4da-6f1383a9db2d-kube-api-access-275k2\") pod \"test-curl-7408-0caf3b\" (UID: \"8e1ec39f-1c6a-4442-b4da-6f1383a9db2d\") " pod="opendatahub/test-curl-7408-0caf3b"
Oct 30 19:04:17.018069 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:17.018001    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7408-0caf3b"
Oct 30 19:04:17.309919 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:17.309887    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7408-0caf3b"]
Oct 30 19:04:17.310586 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:04:17.310558    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e1ec39f_1c6a_4442_b4da_6f1383a9db2d.slice/crio-fe5f4b4a445445e1270311e0ccdecc8bac7cf13cc537739d9a76908d53f8187e WatchSource:0}: Error finding container fe5f4b4a445445e1270311e0ccdecc8bac7cf13cc537739d9a76908d53f8187e: Status 404 returned error can't find the container with id fe5f4b4a445445e1270311e0ccdecc8bac7cf13cc537739d9a76908d53f8187e
Oct 30 19:04:17.591450 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:17.591414    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7408-0caf3b" event={"ID":"8e1ec39f-1c6a-4442-b4da-6f1383a9db2d","Type":"ContainerStarted","Data":"3de6f30e3d9e68253e629774e66b4870068b95ff854f1726de0c9268a43c6290"}
Oct 30 19:04:17.591626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:17.591467    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7408-0caf3b" event={"ID":"8e1ec39f-1c6a-4442-b4da-6f1383a9db2d","Type":"ContainerStarted","Data":"fe5f4b4a445445e1270311e0ccdecc8bac7cf13cc537739d9a76908d53f8187e"}
Oct 30 19:04:17.611975 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:17.611928    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7408-0caf3b" podStartSLOduration=1.6119152799999998 podStartE2EDuration="1.61191528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:17.610765313 +0000 UTC m=+1871.479183644" watchObservedRunningTime="2026-10-30 19:04:17.61191528 +0000 UTC m=+1871.480333609"
Oct 30 19:04:18.389231 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:18.389199    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7408-0caf3b"]
Oct 30 19:04:18.392939 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:18.392911    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7408-0caf3b"]
Oct 30 19:04:19.421313 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.421279    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"]
Oct 30 19:04:19.421737 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.421465    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy" containerID="cri-o://4b65d792d8ac6aad87fc4063ff44ba654a52752f20760c1c5530b7b4176fe912" gracePeriod=30
Oct 30 19:04:19.606535 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.606493    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7408-0caf3b" podUID="8e1ec39f-1c6a-4442-b4da-6f1383a9db2d" containerName="test-curl-7408-0caf3b" containerID="cri-o://3de6f30e3d9e68253e629774e66b4870068b95ff854f1726de0c9268a43c6290" gracePeriod=2
Oct 30 19:04:19.609767 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:04:19.609736    4372 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7408-0caf3b\" is forbidden: User \"system:node:ip-10-0-142-153.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-153.us-east-2.compute.internal' and this object" podUID="8e1ec39f-1c6a-4442-b4da-6f1383a9db2d" pod="opendatahub/test-curl-7408-0caf3b"
Oct 30 19:04:19.778215 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.778190    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 19:04:19.806740 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:04:19.806703    4372 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7408-0caf3b\" is forbidden: User \"system:node:ip-10-0-142-153.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-153.us-east-2.compute.internal' and this object" podUID="8e1ec39f-1c6a-4442-b4da-6f1383a9db2d" pod="opendatahub/test-curl-7408-0caf3b"
Oct 30 19:04:19.899635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.899604    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-kube-api-access-r8d7p\" (UniqueName: \"kubernetes.io/projected/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-kube-api-access-r8d7p\") pod \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") "
Oct 30 19:04:19.899798 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.899642    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-podinfo\") pod \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") "
Oct 30 19:04:19.899798 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.899674    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-workload-certs\") pod \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") "
Oct 30 19:04:19.899926 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.899826    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-data\") pod \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") "
Oct 30 19:04:19.899926 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.899869    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istiod-ca-cert\") pod \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") "
Oct 30 19:04:19.899926 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.899906    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-token\" (UniqueName: \"kubernetes.io/projected/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-token\") pod \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") "
Oct 30 19:04:19.900179 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.899994    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-credential-socket\") pod \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") "
Oct 30 19:04:19.900179 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.900078    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-workload-certs" pod "76b49aa4-fbad-4b2d-91b0-bece668fbfd3" (UID: "76b49aa4-fbad-4b2d-91b0-bece668fbfd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:19.900179 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.900090    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-workload-socket\") pod \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") "
Oct 30 19:04:19.900179 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.900142    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-data" pod "76b49aa4-fbad-4b2d-91b0-bece668fbfd3" (UID: "76b49aa4-fbad-4b2d-91b0-bece668fbfd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:19.900383 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.900212    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-envoy\") pod \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\" (UID: \"76b49aa4-fbad-4b2d-91b0-bece668fbfd3\") "
Oct 30 19:04:19.900383 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.900341    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-workload-socket" pod "76b49aa4-fbad-4b2d-91b0-bece668fbfd3" (UID: "76b49aa4-fbad-4b2d-91b0-bece668fbfd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:19.900494 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.900466    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-credential-socket" pod "76b49aa4-fbad-4b2d-91b0-bece668fbfd3" (UID: "76b49aa4-fbad-4b2d-91b0-bece668fbfd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:19.900558 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.900537    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istiod-ca-cert" pod "76b49aa4-fbad-4b2d-91b0-bece668fbfd3" (UID: "76b49aa4-fbad-4b2d-91b0-bece668fbfd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:19.901254 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.901227    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:19.901380 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.901257    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:19.901380 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.901272    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:19.901380 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.901289    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:19.901380 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.901304    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:19.902526 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.902504    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-token" pod "76b49aa4-fbad-4b2d-91b0-bece668fbfd3" (UID: "76b49aa4-fbad-4b2d-91b0-bece668fbfd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:19.902706 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.902688    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-envoy" pod "76b49aa4-fbad-4b2d-91b0-bece668fbfd3" (UID: "76b49aa4-fbad-4b2d-91b0-bece668fbfd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:19.902787 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.902736    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-kube-api-access-r8d7p" pod "76b49aa4-fbad-4b2d-91b0-bece668fbfd3" (UID: "76b49aa4-fbad-4b2d-91b0-bece668fbfd3"). InnerVolumeSpecName "kube-api-access-r8d7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:19.902787 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:19.902743    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-podinfo" pod "76b49aa4-fbad-4b2d-91b0-bece668fbfd3" (UID: "76b49aa4-fbad-4b2d-91b0-bece668fbfd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:04:20.009600 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.008513    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:20.009600 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.008552    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:20.009600 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.008567    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8d7p\" (UniqueName: \"kubernetes.io/projected/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-kube-api-access-r8d7p\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:20.009600 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.008582    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b49aa4-fbad-4b2d-91b0-bece668fbfd3-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:20.367693 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.367628    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng" podUID="3c97b261-f060-45a3-95e5-a6f924b51047" containerName="main" containerID="cri-o://40b1737878a157569517bc689ba364bf965a9b09aa31eb7faa06159fd73792bf" gracePeriod=45
Oct 30 19:04:20.525733 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.525694    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng" podUID="3c97b261-f060-45a3-95e5-a6f924b51047" containerName="main" probeResult="failure" output="Get \"https://10.133.0.170:8000/ready\": dial tcp 10.133.0.170:8000: connect: connection refused"
Oct 30 19:04:20.531702 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.531655    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"]
Oct 30 19:04:20.532924 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.532846    4372 cpu_manager.go:414] "containerMap: removing container" podUID="8e1ec39f-1c6a-4442-b4da-6f1383a9db2d" containerName="test-curl-7408-0caf3b"
Oct 30 19:04:20.532924 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.532882    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1ec39f-1c6a-4442-b4da-6f1383a9db2d" containerName="test-curl-7408-0caf3b"
Oct 30 19:04:20.532924 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.532910    4372 cpu_manager.go:414] "containerMap: removing container" podUID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" containerName="istio-proxy"
Oct 30 19:04:20.532924 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.532919    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" containerName="istio-proxy"
Oct 30 19:04:20.533265 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.533156    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1ec39f-1c6a-4442-b4da-6f1383a9db2d" containerName="test-curl-7408-0caf3b"
Oct 30 19:04:20.533265 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.533174    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" containerName="istio-proxy"
Oct 30 19:04:20.537533 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.537512    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.547249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.547228    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-21d896-openshift-default-dockercfg-xdx6m\""
Oct 30 19:04:20.561369 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.561340    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"]
Oct 30 19:04:20.616555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.616521    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6ca59e-1c18-468a-8ba1-23aca6351823-istiod-ca-cert\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.616828 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.616806    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-workload-certs\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.616942 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.616845    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-credential-socket\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.617012 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.616970    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwkcd\" (UniqueName: \"kubernetes.io/projected/8b6ca59e-1c18-468a-8ba1-23aca6351823-kube-api-access-rwkcd\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.617068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.617008    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-envoy\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.617068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.617039    4372 generic.go:356] "Generic (PLEG): container finished" podID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" containerID="6f0a9e869f2f9e60e06c8c2e6e5ee23a60ae7299fafbc645bb68f042cbc3d4fd" exitCode=0
Oct 30 19:04:20.617234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.617098    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-data\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.617234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.617206    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-token\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.617341 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.617248    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" event={"ID":"76b49aa4-fbad-4b2d-91b0-bece668fbfd3","Type":"ContainerDied","Data":"6f0a9e869f2f9e60e06c8c2e6e5ee23a60ae7299fafbc645bb68f042cbc3d4fd"}
Oct 30 19:04:20.617341 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.617263    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-workload-socket\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.617341 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.617289    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk" event={"ID":"76b49aa4-fbad-4b2d-91b0-bece668fbfd3","Type":"ContainerDied","Data":"6451c99617ee8a1da18ca730ac6c52597ed6f5fdb667b4b127d4cb992426396b"}
Oct 30 19:04:20.617341 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.617316    4372 scope.go:122] "RemoveContainer" containerID="6f0a9e869f2f9e60e06c8c2e6e5ee23a60ae7299fafbc645bb68f042cbc3d4fd"
Oct 30 19:04:20.617545 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.617350    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-podinfo\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.617545 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.617438    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"
Oct 30 19:04:20.621429 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.621402    4372 generic.go:356] "Generic (PLEG): container finished" podID="3c97b261-f060-45a3-95e5-a6f924b51047" containerID="40b1737878a157569517bc689ba364bf965a9b09aa31eb7faa06159fd73792bf" exitCode=0
Oct 30 19:04:20.621540 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.621468    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng" event={"ID":"3c97b261-f060-45a3-95e5-a6f924b51047","Type":"ContainerDied","Data":"40b1737878a157569517bc689ba364bf965a9b09aa31eb7faa06159fd73792bf"}
Oct 30 19:04:20.631142 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.631056    4372 scope.go:122] "RemoveContainer" containerID="6f0a9e869f2f9e60e06c8c2e6e5ee23a60ae7299fafbc645bb68f042cbc3d4fd"
Oct 30 19:04:20.631971 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:04:20.631549    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0a9e869f2f9e60e06c8c2e6e5ee23a60ae7299fafbc645bb68f042cbc3d4fd\": container with ID starting with 6f0a9e869f2f9e60e06c8c2e6e5ee23a60ae7299fafbc645bb68f042cbc3d4fd not found: ID does not exist" containerID="6f0a9e869f2f9e60e06c8c2e6e5ee23a60ae7299fafbc645bb68f042cbc3d4fd"
Oct 30 19:04:20.631971 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.631585    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0a9e869f2f9e60e06c8c2e6e5ee23a60ae7299fafbc645bb68f042cbc3d4fd"} err="failed to get container status \"6f0a9e869f2f9e60e06c8c2e6e5ee23a60ae7299fafbc645bb68f042cbc3d4fd\": rpc error: code = NotFound desc = could not find container \"6f0a9e869f2f9e60e06c8c2e6e5ee23a60ae7299fafbc645bb68f042cbc3d4fd\": container with ID starting with 6f0a9e869f2f9e60e06c8c2e6e5ee23a60ae7299fafbc645bb68f042cbc3d4fd not found: ID does not exist"
Oct 30 19:04:20.648158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.648112    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"]
Oct 30 19:04:20.654635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.654611    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-4fqfk"]
Oct 30 19:04:20.696762 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.696740    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 19:04:20.718519 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.718498    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-workload-certs\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.718613 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.718530    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-credential-socket\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.718664 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.718614    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwkcd\" (UniqueName: \"kubernetes.io/projected/8b6ca59e-1c18-468a-8ba1-23aca6351823-kube-api-access-rwkcd\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.718664 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.718643    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-envoy\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.718775 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.718701    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-data\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.718775 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.718759    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-token\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.718895 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.718810    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-workload-socket\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.718895 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.718878    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-podinfo\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.719012 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.718929    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6ca59e-1c18-468a-8ba1-23aca6351823-istiod-ca-cert\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.719012 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.718948    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-workload-certs\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.719012 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.718991    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-credential-socket\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.719223 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.719206    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-data\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.719337 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.719314    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-workload-socket\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.719924 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.719900    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6ca59e-1c18-468a-8ba1-23aca6351823-istiod-ca-cert\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.721287 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.721270    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-envoy\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.721514 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.721493    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-podinfo\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:20.947006 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.946964    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-home\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-home\") pod \"3c97b261-f060-45a3-95e5-a6f924b51047\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") "
Oct 30 19:04:20.947278 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.947118    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c97b261-f060-45a3-95e5-a6f924b51047-kube-api-access-654ht\" (UniqueName: \"kubernetes.io/projected/3c97b261-f060-45a3-95e5-a6f924b51047-kube-api-access-654ht\") pod \"3c97b261-f060-45a3-95e5-a6f924b51047\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") "
Oct 30 19:04:20.947278 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.947242    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-model-cache\") pod \"3c97b261-f060-45a3-95e5-a6f924b51047\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") "
Oct 30 19:04:20.947448 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.947339    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-tmp-dir\") pod \"3c97b261-f060-45a3-95e5-a6f924b51047\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") "
Oct 30 19:04:20.947448 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.947373    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-dshm\") pod \"3c97b261-f060-45a3-95e5-a6f924b51047\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") "
Oct 30 19:04:20.947558 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.947499    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c97b261-f060-45a3-95e5-a6f924b51047-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c97b261-f060-45a3-95e5-a6f924b51047-tls-certs\") pod \"3c97b261-f060-45a3-95e5-a6f924b51047\" (UID: \"3c97b261-f060-45a3-95e5-a6f924b51047\") "
Oct 30 19:04:21.008197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.982245    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-home" pod "3c97b261-f060-45a3-95e5-a6f924b51047" (UID: "3c97b261-f060-45a3-95e5-a6f924b51047"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:21.008197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:20.993571    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-tmp-dir" pod "3c97b261-f060-45a3-95e5-a6f924b51047" (UID: "3c97b261-f060-45a3-95e5-a6f924b51047"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:21.042158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.024610    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-model-cache" pod "3c97b261-f060-45a3-95e5-a6f924b51047" (UID: "3c97b261-f060-45a3-95e5-a6f924b51047"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:21.083552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.080998    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-dshm" pod "3c97b261-f060-45a3-95e5-a6f924b51047" (UID: "3c97b261-f060-45a3-95e5-a6f924b51047"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:21.083552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.081496    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:21.083552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.081536    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:21.083552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.081555    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:21.083552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.081568    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c97b261-f060-45a3-95e5-a6f924b51047-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:21.083552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.083427    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c97b261-f060-45a3-95e5-a6f924b51047-kube-api-access-654ht" pod "3c97b261-f060-45a3-95e5-a6f924b51047" (UID: "3c97b261-f060-45a3-95e5-a6f924b51047"). InnerVolumeSpecName "kube-api-access-654ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:21.083552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.083522    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c97b261-f060-45a3-95e5-a6f924b51047-tls-certs" pod "3c97b261-f060-45a3-95e5-a6f924b51047" (UID: "3c97b261-f060-45a3-95e5-a6f924b51047"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:21.103643 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.093956    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwkcd\" (UniqueName: \"kubernetes.io/projected/8b6ca59e-1c18-468a-8ba1-23aca6351823-kube-api-access-rwkcd\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:21.103643 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.098717    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-token\") pod \"e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") " pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:21.172145 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.168415    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:21.185453 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.184741    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c97b261-f060-45a3-95e5-a6f924b51047-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:21.185453 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.184765    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-654ht\" (UniqueName: \"kubernetes.io/projected/3c97b261-f060-45a3-95e5-a6f924b51047-kube-api-access-654ht\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:21.471169 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:04:21.471119    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6ca59e_1c18_468a_8ba1_23aca6351823.slice/crio-d449e9f704848f0e697663fec40f49cd439f1053ee440911606cee6cd213bd23 WatchSource:0}: Error finding container d449e9f704848f0e697663fec40f49cd439f1053ee440911606cee6cd213bd23: Status 404 returned error can't find the container with id d449e9f704848f0e697663fec40f49cd439f1053ee440911606cee6cd213bd23
Oct 30 19:04:21.471542 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.471517    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"]
Oct 30 19:04:21.473862 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.473835    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:04:21.473944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.473893    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:04:21.473944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.473929    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:04:21.633352 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.633323    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"
Oct 30 19:04:21.633747 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.633353    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng" event={"ID":"3c97b261-f060-45a3-95e5-a6f924b51047","Type":"ContainerDied","Data":"f47cf9dbcad7f9270ff336abee97040b6bfd6a102cacf7dd1eca979901714321"}
Oct 30 19:04:21.633747 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.633413    4372 scope.go:122] "RemoveContainer" containerID="40b1737878a157569517bc689ba364bf965a9b09aa31eb7faa06159fd73792bf"
Oct 30 19:04:21.635474 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.635449    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k" event={"ID":"8b6ca59e-1c18-468a-8ba1-23aca6351823","Type":"ContainerStarted","Data":"ca4ddc6ed25aa2e36a7bb17470f3deea14e1f50bde13517ccbfd8a2b096691ee"}
Oct 30 19:04:21.635581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.635499    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k" event={"ID":"8b6ca59e-1c18-468a-8ba1-23aca6351823","Type":"ContainerStarted","Data":"d449e9f704848f0e697663fec40f49cd439f1053ee440911606cee6cd213bd23"}
Oct 30 19:04:21.667523 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.666043    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k" podStartSLOduration=1.666024487 podStartE2EDuration="1.666024487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:21.662215653 +0000 UTC m=+1875.530633984" watchObservedRunningTime="2026-10-30 19:04:21.666024487 +0000 UTC m=+1875.534442818"
Oct 30 19:04:21.696835 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.696802    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"]
Oct 30 19:04:21.700800 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.700773    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a962a/auth-test-model-0a962a-kserve-7f75465669-rs2ng"]
Oct 30 19:04:21.928538 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:21.928510    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7408-0caf3b"
Oct 30 19:04:22.009296 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:22.009267    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1ec39f-1c6a-4442-b4da-6f1383a9db2d-kube-api-access-275k2\" (UniqueName: \"kubernetes.io/projected/8e1ec39f-1c6a-4442-b4da-6f1383a9db2d-kube-api-access-275k2\") pod \"8e1ec39f-1c6a-4442-b4da-6f1383a9db2d\" (UID: \"8e1ec39f-1c6a-4442-b4da-6f1383a9db2d\") "
Oct 30 19:04:22.011523 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:22.011500    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1ec39f-1c6a-4442-b4da-6f1383a9db2d-kube-api-access-275k2" pod "8e1ec39f-1c6a-4442-b4da-6f1383a9db2d" (UID: "8e1ec39f-1c6a-4442-b4da-6f1383a9db2d"). InnerVolumeSpecName "kube-api-access-275k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:22.115147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:22.113699    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-275k2\" (UniqueName: \"kubernetes.io/projected/8e1ec39f-1c6a-4442-b4da-6f1383a9db2d-kube-api-access-275k2\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:22.169813 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:22.169734    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:22.176706 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:22.176676    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:22.267045 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:22.267016    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c97b261-f060-45a3-95e5-a6f924b51047" path="/var/lib/kubelet/pods/3c97b261-f060-45a3-95e5-a6f924b51047/volumes"
Oct 30 19:04:22.267574 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:22.267559    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b49aa4-fbad-4b2d-91b0-bece668fbfd3" path="/var/lib/kubelet/pods/76b49aa4-fbad-4b2d-91b0-bece668fbfd3/volumes"
Oct 30 19:04:22.267931 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:22.267918    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1ec39f-1c6a-4442-b4da-6f1383a9db2d" path="/var/lib/kubelet/pods/8e1ec39f-1c6a-4442-b4da-6f1383a9db2d/volumes"
Oct 30 19:04:22.646245 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:22.646212    4372 generic.go:356] "Generic (PLEG): container finished" podID="8e1ec39f-1c6a-4442-b4da-6f1383a9db2d" containerID="3de6f30e3d9e68253e629774e66b4870068b95ff854f1726de0c9268a43c6290" exitCode=137
Oct 30 19:04:22.646669 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:22.646309    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7408-0caf3b"
Oct 30 19:04:22.646669 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:22.646368    4372 scope.go:122] "RemoveContainer" containerID="3de6f30e3d9e68253e629774e66b4870068b95ff854f1726de0c9268a43c6290"
Oct 30 19:04:22.646838 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:22.646803    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:22.647840 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:22.647817    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:04:22.655545 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:22.655528    4372 scope.go:122] "RemoveContainer" containerID="3de6f30e3d9e68253e629774e66b4870068b95ff854f1726de0c9268a43c6290"
Oct 30 19:04:22.655819 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:04:22.655800    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de6f30e3d9e68253e629774e66b4870068b95ff854f1726de0c9268a43c6290\": container with ID starting with 3de6f30e3d9e68253e629774e66b4870068b95ff854f1726de0c9268a43c6290 not found: ID does not exist" containerID="3de6f30e3d9e68253e629774e66b4870068b95ff854f1726de0c9268a43c6290"
Oct 30 19:04:22.655891 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:22.655828    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de6f30e3d9e68253e629774e66b4870068b95ff854f1726de0c9268a43c6290"} err="failed to get container status \"3de6f30e3d9e68253e629774e66b4870068b95ff854f1726de0c9268a43c6290\": rpc error: code = NotFound desc = could not find container \"3de6f30e3d9e68253e629774e66b4870068b95ff854f1726de0c9268a43c6290\": container with ID starting with 3de6f30e3d9e68253e629774e66b4870068b95ff854f1726de0c9268a43c6290 not found: ID does not exist"
Oct 30 19:04:24.667332 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.667302    4372 generic.go:356] "Generic (PLEG): container finished" podID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerID="4b65d792d8ac6aad87fc4063ff44ba654a52752f20760c1c5530b7b4176fe912" exitCode=0
Oct 30 19:04:24.667803 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.667383    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" event={"ID":"8795b2cd-8ffe-4582-904d-2ae5751a2e09","Type":"ContainerDied","Data":"4b65d792d8ac6aad87fc4063ff44ba654a52752f20760c1c5530b7b4176fe912"}
Oct 30 19:04:24.698519 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.698460    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 19:04:24.797830 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.797788    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk" podUID="4549f795-bff1-43c4-8f1a-5493b5f975d8" containerName="main" containerID="cri-o://51a93e2a5e29772a0eba742cbd35f1c134529a073225b19dd580d990a5383841" gracePeriod=45
Oct 30 19:04:24.870762 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.870723    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-workload-certs\") pod \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") "
Oct 30 19:04:24.870935 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.870904    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8795b2cd-8ffe-4582-904d-2ae5751a2e09-kube-api-access-58ssl\" (UniqueName: \"kubernetes.io/projected/8795b2cd-8ffe-4582-904d-2ae5751a2e09-kube-api-access-58ssl\") pod \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") "
Oct 30 19:04:24.871013 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.870994    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-workload-certs" pod "8795b2cd-8ffe-4582-904d-2ae5751a2e09" (UID: "8795b2cd-8ffe-4582-904d-2ae5751a2e09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.871080 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.871021    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-workload-socket\") pod \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") "
Oct 30 19:04:24.871254 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.871224    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-podinfo\") pod \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") "
Oct 30 19:04:24.871362 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.871290    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-token\" (UniqueName: \"kubernetes.io/projected/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-token\") pod \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") "
Oct 30 19:04:24.871362 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.871317    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-workload-socket" pod "8795b2cd-8ffe-4582-904d-2ae5751a2e09" (UID: "8795b2cd-8ffe-4582-904d-2ae5751a2e09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.871486 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.871370    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-envoy\") pod \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") "
Oct 30 19:04:24.871486 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.871442    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-data\") pod \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") "
Oct 30 19:04:24.871486 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.871470    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istiod-ca-cert\") pod \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") "
Oct 30 19:04:24.871628 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.871523    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-credential-socket\") pod \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\" (UID: \"8795b2cd-8ffe-4582-904d-2ae5751a2e09\") "
Oct 30 19:04:24.872578 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.872022    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-data" pod "8795b2cd-8ffe-4582-904d-2ae5751a2e09" (UID: "8795b2cd-8ffe-4582-904d-2ae5751a2e09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.872578 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.872281    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-credential-socket" pod "8795b2cd-8ffe-4582-904d-2ae5751a2e09" (UID: "8795b2cd-8ffe-4582-904d-2ae5751a2e09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.872578 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.872354    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istiod-ca-cert" pod "8795b2cd-8ffe-4582-904d-2ae5751a2e09" (UID: "8795b2cd-8ffe-4582-904d-2ae5751a2e09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:24.872578 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.872488    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.872578 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.872510    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.872578 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.872524    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.872578 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.872537    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.879735 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.879707    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8795b2cd-8ffe-4582-904d-2ae5751a2e09-kube-api-access-58ssl" pod "8795b2cd-8ffe-4582-904d-2ae5751a2e09" (UID: "8795b2cd-8ffe-4582-904d-2ae5751a2e09"). InnerVolumeSpecName "kube-api-access-58ssl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:24.879855 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.879799    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-podinfo" pod "8795b2cd-8ffe-4582-904d-2ae5751a2e09" (UID: "8795b2cd-8ffe-4582-904d-2ae5751a2e09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:04:24.879855 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.879838    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-envoy" pod "8795b2cd-8ffe-4582-904d-2ae5751a2e09" (UID: "8795b2cd-8ffe-4582-904d-2ae5751a2e09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.879995 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.879934    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-token" pod "8795b2cd-8ffe-4582-904d-2ae5751a2e09" (UID: "8795b2cd-8ffe-4582-904d-2ae5751a2e09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:24.974370 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.974299    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.974370 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.974328    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.974370 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.974338    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.974370 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.974347    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8795b2cd-8ffe-4582-904d-2ae5751a2e09-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.974370 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:24.974356    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58ssl\" (UniqueName: \"kubernetes.io/projected/8795b2cd-8ffe-4582-904d-2ae5751a2e09-kube-api-access-58ssl\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:25.151521 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.151495    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:04:25.278692 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.278617    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-home\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-home\") pod \"4549f795-bff1-43c4-8f1a-5493b5f975d8\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") "
Oct 30 19:04:25.278692 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.278655    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4549f795-bff1-43c4-8f1a-5493b5f975d8-kube-api-access-ngkf7\" (UniqueName: \"kubernetes.io/projected/4549f795-bff1-43c4-8f1a-5493b5f975d8-kube-api-access-ngkf7\") pod \"4549f795-bff1-43c4-8f1a-5493b5f975d8\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") "
Oct 30 19:04:25.278892 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.278836    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-model-cache\") pod \"4549f795-bff1-43c4-8f1a-5493b5f975d8\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") "
Oct 30 19:04:25.278892 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.278881    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-home" pod "4549f795-bff1-43c4-8f1a-5493b5f975d8" (UID: "4549f795-bff1-43c4-8f1a-5493b5f975d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:25.278978 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.278964    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-tmp-dir\") pod \"4549f795-bff1-43c4-8f1a-5493b5f975d8\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") "
Oct 30 19:04:25.279053 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.279034    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4549f795-bff1-43c4-8f1a-5493b5f975d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/4549f795-bff1-43c4-8f1a-5493b5f975d8-tls-certs\") pod \"4549f795-bff1-43c4-8f1a-5493b5f975d8\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") "
Oct 30 19:04:25.279053 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.279042    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-model-cache" pod "4549f795-bff1-43c4-8f1a-5493b5f975d8" (UID: "4549f795-bff1-43c4-8f1a-5493b5f975d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:25.279227 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.279089    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-dshm\") pod \"4549f795-bff1-43c4-8f1a-5493b5f975d8\" (UID: \"4549f795-bff1-43c4-8f1a-5493b5f975d8\") "
Oct 30 19:04:25.279289 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.279229    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-tmp-dir" pod "4549f795-bff1-43c4-8f1a-5493b5f975d8" (UID: "4549f795-bff1-43c4-8f1a-5493b5f975d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:25.279731 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.279632    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:25.279731 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.279660    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:25.279731 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.279672    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:25.281291 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.281259    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4549f795-bff1-43c4-8f1a-5493b5f975d8-kube-api-access-ngkf7" pod "4549f795-bff1-43c4-8f1a-5493b5f975d8" (UID: "4549f795-bff1-43c4-8f1a-5493b5f975d8"). InnerVolumeSpecName "kube-api-access-ngkf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:25.281385 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.281293    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-dshm" pod "4549f795-bff1-43c4-8f1a-5493b5f975d8" (UID: "4549f795-bff1-43c4-8f1a-5493b5f975d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:25.281385 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.281345    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4549f795-bff1-43c4-8f1a-5493b5f975d8-tls-certs" pod "4549f795-bff1-43c4-8f1a-5493b5f975d8" (UID: "4549f795-bff1-43c4-8f1a-5493b5f975d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:25.381708 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.381675    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4549f795-bff1-43c4-8f1a-5493b5f975d8-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:25.381708 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.381710    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4549f795-bff1-43c4-8f1a-5493b5f975d8-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:25.381888 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.381728    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngkf7\" (UniqueName: \"kubernetes.io/projected/4549f795-bff1-43c4-8f1a-5493b5f975d8-kube-api-access-ngkf7\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:25.444062 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.444029    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"]
Oct 30 19:04:25.444682 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.444668    4372 cpu_manager.go:414] "containerMap: removing container" podUID="3c97b261-f060-45a3-95e5-a6f924b51047" containerName="main"
Oct 30 19:04:25.444743 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.444685    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c97b261-f060-45a3-95e5-a6f924b51047" containerName="main"
Oct 30 19:04:25.444743 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.444708    4372 cpu_manager.go:414] "containerMap: removing container" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy"
Oct 30 19:04:25.444743 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.444714    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy"
Oct 30 19:04:25.444743 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.444743    4372 cpu_manager.go:414] "containerMap: removing container" podUID="4549f795-bff1-43c4-8f1a-5493b5f975d8" containerName="main"
Oct 30 19:04:25.444878 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.444749    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4549f795-bff1-43c4-8f1a-5493b5f975d8" containerName="main"
Oct 30 19:04:25.444878 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.444844    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="4549f795-bff1-43c4-8f1a-5493b5f975d8" containerName="main"
Oct 30 19:04:25.444878 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.444858    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c97b261-f060-45a3-95e5-a6f924b51047" containerName="main"
Oct 30 19:04:25.444878 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.444865    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" containerName="istio-proxy"
Oct 30 19:04:25.448620 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.448602    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"
Oct 30 19:04:25.454856 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.454834    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-21d896\""
Oct 30 19:04:25.470796 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.470750    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"]
Oct 30 19:04:25.476479 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.476455    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"]
Oct 30 19:04:25.476615 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.476566    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"
Oct 30 19:04:25.490603 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.490549    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-21d896-dockercfg-xj2dg\""
Oct 30 19:04:25.498406 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.498380    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"]
Oct 30 19:04:25.584752 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.584723    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5txg\" (UniqueName: \"kubernetes.io/projected/0e7d26de-2772-4819-88ed-4ab9dc24bba0-kube-api-access-d5txg\") pod \"payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9\" (UID: \"0e7d26de-2772-4819-88ed-4ab9dc24bba0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"
Oct 30 19:04:25.584904 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.584769    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7d26de-2772-4819-88ed-4ab9dc24bba0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9\" (UID: \"0e7d26de-2772-4819-88ed-4ab9dc24bba0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"
Oct 30 19:04:25.584998 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.584977    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whct4\" (UniqueName: \"kubernetes.io/projected/008d989a-f0e1-4611-8206-0986d02786a3-kube-api-access-whct4\") pod \"payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6\" (UID: \"008d989a-f0e1-4611-8206-0986d02786a3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"
Oct 30 19:04:25.585056 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.585043    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/008d989a-f0e1-4611-8206-0986d02786a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6\" (UID: \"008d989a-f0e1-4611-8206-0986d02786a3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"
Oct 30 19:04:25.693118 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.691299    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7d26de-2772-4819-88ed-4ab9dc24bba0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9\" (UID: \"0e7d26de-2772-4819-88ed-4ab9dc24bba0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"
Oct 30 19:04:25.693118 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.691623    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whct4\" (UniqueName: \"kubernetes.io/projected/008d989a-f0e1-4611-8206-0986d02786a3-kube-api-access-whct4\") pod \"payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6\" (UID: \"008d989a-f0e1-4611-8206-0986d02786a3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"
Oct 30 19:04:25.693118 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.691728    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/008d989a-f0e1-4611-8206-0986d02786a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6\" (UID: \"008d989a-f0e1-4611-8206-0986d02786a3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"
Oct 30 19:04:25.693118 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.691833    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5txg\" (UniqueName: \"kubernetes.io/projected/0e7d26de-2772-4819-88ed-4ab9dc24bba0-kube-api-access-d5txg\") pod \"payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9\" (UID: \"0e7d26de-2772-4819-88ed-4ab9dc24bba0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"
Oct 30 19:04:25.693118 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.692991    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7d26de-2772-4819-88ed-4ab9dc24bba0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9\" (UID: \"0e7d26de-2772-4819-88ed-4ab9dc24bba0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"
Oct 30 19:04:25.693118 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.692991    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/008d989a-f0e1-4611-8206-0986d02786a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6\" (UID: \"008d989a-f0e1-4611-8206-0986d02786a3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"
Oct 30 19:04:25.759148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.755935    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5txg\" (UniqueName: \"kubernetes.io/projected/0e7d26de-2772-4819-88ed-4ab9dc24bba0-kube-api-access-d5txg\") pod \"payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9\" (UID: \"0e7d26de-2772-4819-88ed-4ab9dc24bba0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"
Oct 30 19:04:25.759148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.756660    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whct4\" (UniqueName: \"kubernetes.io/projected/008d989a-f0e1-4611-8206-0986d02786a3-kube-api-access-whct4\") pod \"payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6\" (UID: \"008d989a-f0e1-4611-8206-0986d02786a3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"
Oct 30 19:04:25.806422 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.781542    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"
Oct 30 19:04:25.806422 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.796290    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"
Oct 30 19:04:25.869634 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.869582    4372 generic.go:356] "Generic (PLEG): container finished" podID="4549f795-bff1-43c4-8f1a-5493b5f975d8" containerID="51a93e2a5e29772a0eba742cbd35f1c134529a073225b19dd580d990a5383841" exitCode=0
Oct 30 19:04:25.869913 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.869848    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk" event={"ID":"4549f795-bff1-43c4-8f1a-5493b5f975d8","Type":"ContainerDied","Data":"51a93e2a5e29772a0eba742cbd35f1c134529a073225b19dd580d990a5383841"}
Oct 30 19:04:25.869913 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.869910    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk" event={"ID":"4549f795-bff1-43c4-8f1a-5493b5f975d8","Type":"ContainerDied","Data":"f3e059408c0445a2c98db46f11947cd32957ad0d35951ba6a5b85adb83853772"}
Oct 30 19:04:25.870079 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.869940    4372 scope.go:122] "RemoveContainer" containerID="51a93e2a5e29772a0eba742cbd35f1c134529a073225b19dd580d990a5383841"
Oct 30 19:04:25.873306 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.870241    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"
Oct 30 19:04:25.925287 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.916611    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds" event={"ID":"8795b2cd-8ffe-4582-904d-2ae5751a2e09","Type":"ContainerDied","Data":"6f7953e6387e7e2f2d20abda44d91d44d0b1337b8c4745902145665305e4a764"}
Oct 30 19:04:25.925287 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.916965    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"
Oct 30 19:04:25.986349 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.986322    4372 scope.go:122] "RemoveContainer" containerID="51a93e2a5e29772a0eba742cbd35f1c134529a073225b19dd580d990a5383841"
Oct 30 19:04:25.986816 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:04:25.986785    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a93e2a5e29772a0eba742cbd35f1c134529a073225b19dd580d990a5383841\": container with ID starting with 51a93e2a5e29772a0eba742cbd35f1c134529a073225b19dd580d990a5383841 not found: ID does not exist" containerID="51a93e2a5e29772a0eba742cbd35f1c134529a073225b19dd580d990a5383841"
Oct 30 19:04:25.986928 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.986832    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a93e2a5e29772a0eba742cbd35f1c134529a073225b19dd580d990a5383841"} err="failed to get container status \"51a93e2a5e29772a0eba742cbd35f1c134529a073225b19dd580d990a5383841\": rpc error: code = NotFound desc = could not find container \"51a93e2a5e29772a0eba742cbd35f1c134529a073225b19dd580d990a5383841\": container with ID starting with 51a93e2a5e29772a0eba742cbd35f1c134529a073225b19dd580d990a5383841 not found: ID does not exist"
Oct 30 19:04:25.986928 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:25.986860    4372 scope.go:122] "RemoveContainer" containerID="4b65d792d8ac6aad87fc4063ff44ba654a52752f20760c1c5530b7b4176fe912"
Oct 30 19:04:26.008512 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.008483    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"]
Oct 30 19:04:26.015287 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.015263    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-85dds"]
Oct 30 19:04:26.046078 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.046049    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"]
Oct 30 19:04:26.052888 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.052860    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ab7df0/auth-test-model-ab7df0-kserve-89fbfff7-w2gwk"]
Oct 30 19:04:26.264743 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:04:26.264715    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7d26de_2772_4819_88ed_4ab9dc24bba0.slice/crio-d611958449e0fea77e14c7f3c603e4bcde17c14270199ded0fdd9b8b23f3e6a8 WatchSource:0}: Error finding container d611958449e0fea77e14c7f3c603e4bcde17c14270199ded0fdd9b8b23f3e6a8: Status 404 returned error can't find the container with id d611958449e0fea77e14c7f3c603e4bcde17c14270199ded0fdd9b8b23f3e6a8
Oct 30 19:04:26.270499 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.270467    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4549f795-bff1-43c4-8f1a-5493b5f975d8" path="/var/lib/kubelet/pods/4549f795-bff1-43c4-8f1a-5493b5f975d8/volumes"
Oct 30 19:04:26.270979 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.270963    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8795b2cd-8ffe-4582-904d-2ae5751a2e09" path="/var/lib/kubelet/pods/8795b2cd-8ffe-4582-904d-2ae5751a2e09/volumes"
Oct 30 19:04:26.271299 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.271277    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"]
Oct 30 19:04:26.284385 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.284193    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"]
Oct 30 19:04:26.288671 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:04:26.288644    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008d989a_f0e1_4611_8206_0986d02786a3.slice/crio-8eaa907c489797f059f8e63d5a7cb89bf4c4636c9b92263b87ed5a0821c2450d WatchSource:0}: Error finding container 8eaa907c489797f059f8e63d5a7cb89bf4c4636c9b92263b87ed5a0821c2450d: Status 404 returned error can't find the container with id 8eaa907c489797f059f8e63d5a7cb89bf4c4636c9b92263b87ed5a0821c2450d
Oct 30 19:04:26.932472 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.932433    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9" event={"ID":"0e7d26de-2772-4819-88ed-4ab9dc24bba0","Type":"ContainerStarted","Data":"f6f2db1f2205682e93a1fc6e0b33d0b652f71ff0a4d4e8f00801afcb7d31cb7e"}
Oct 30 19:04:26.932995 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.932503    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9" event={"ID":"0e7d26de-2772-4819-88ed-4ab9dc24bba0","Type":"ContainerStarted","Data":"d611958449e0fea77e14c7f3c603e4bcde17c14270199ded0fdd9b8b23f3e6a8"}
Oct 30 19:04:26.932995 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.932552    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"
Oct 30 19:04:26.935812 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.935786    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6" event={"ID":"008d989a-f0e1-4611-8206-0986d02786a3","Type":"ContainerStarted","Data":"e565fdb9a12c0275c3603db5a14ff2944684e3d266e2186cdb7b4e67c8248a4e"}
Oct 30 19:04:26.935961 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.935836    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6" event={"ID":"008d989a-f0e1-4611-8206-0986d02786a3","Type":"ContainerStarted","Data":"8eaa907c489797f059f8e63d5a7cb89bf4c4636c9b92263b87ed5a0821c2450d"}
Oct 30 19:04:26.936038 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.935987    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"
Oct 30 19:04:26.963418 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.963373    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9" podStartSLOduration=1.96335985 podStartE2EDuration="1.96335985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:26.960325208 +0000 UTC m=+1880.828743539" watchObservedRunningTime="2026-10-30 19:04:26.96335985 +0000 UTC m=+1880.831778173"
Oct 30 19:04:26.985968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:26.985925    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6" podStartSLOduration=1.9859147990000001 podStartE2EDuration="1.985914799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:26.983021644 +0000 UTC m=+1880.851439974" watchObservedRunningTime="2026-10-30 19:04:26.985914799 +0000 UTC m=+1880.854333130"
Oct 30 19:04:27.698214 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:04:27.698180    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:04:29.332229 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:29.332201    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"]
Oct 30 19:04:29.332762 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:29.332381    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" podUID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" containerName="istio-proxy" containerID="cri-o://6627b03ea5bee45d11b62146f879ba00305e251de7a63603474c31bec7b886cf" gracePeriod=30
Oct 30 19:04:32.121017 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.120983    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"]
Oct 30 19:04:32.125566 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.125549    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"
Oct 30 19:04:32.136140 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.136098    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"]
Oct 30 19:04:32.136500 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.136481    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-f7c6e3\""
Oct 30 19:04:32.165918 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.165896    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-f7c6e3-dockercfg-wmzpb\""
Oct 30 19:04:32.180268 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.180242    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"]
Oct 30 19:04:32.184660 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.184643    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"
Oct 30 19:04:32.199761 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.199737    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"]
Oct 30 19:04:32.264151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.260863    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4sd5\" (UniqueName: \"kubernetes.io/projected/0fa40c2a-bd44-4da8-95b0-3f68cfd29335-kube-api-access-t4sd5\") pod \"payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8\" (UID: \"0fa40c2a-bd44-4da8-95b0-3f68cfd29335\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"
Oct 30 19:04:32.264151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.261073    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa40c2a-bd44-4da8-95b0-3f68cfd29335-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8\" (UID: \"0fa40c2a-bd44-4da8-95b0-3f68cfd29335\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"
Oct 30 19:04:32.440310 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.435676    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-786kb\" (UniqueName: \"kubernetes.io/projected/becab9bd-103e-4d46-8e6c-fe8c86602dc5-kube-api-access-786kb\") pod \"payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb\" (UID: \"becab9bd-103e-4d46-8e6c-fe8c86602dc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"
Oct 30 19:04:32.440310 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.435851    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4sd5\" (UniqueName: \"kubernetes.io/projected/0fa40c2a-bd44-4da8-95b0-3f68cfd29335-kube-api-access-t4sd5\") pod \"payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8\" (UID: \"0fa40c2a-bd44-4da8-95b0-3f68cfd29335\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"
Oct 30 19:04:32.440310 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.435945    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/becab9bd-103e-4d46-8e6c-fe8c86602dc5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb\" (UID: \"becab9bd-103e-4d46-8e6c-fe8c86602dc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"
Oct 30 19:04:32.440310 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.436007    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa40c2a-bd44-4da8-95b0-3f68cfd29335-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8\" (UID: \"0fa40c2a-bd44-4da8-95b0-3f68cfd29335\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"
Oct 30 19:04:32.440310 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.437058    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa40c2a-bd44-4da8-95b0-3f68cfd29335-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8\" (UID: \"0fa40c2a-bd44-4da8-95b0-3f68cfd29335\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"
Oct 30 19:04:32.564194 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.556303    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-786kb\" (UniqueName: \"kubernetes.io/projected/becab9bd-103e-4d46-8e6c-fe8c86602dc5-kube-api-access-786kb\") pod \"payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb\" (UID: \"becab9bd-103e-4d46-8e6c-fe8c86602dc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"
Oct 30 19:04:32.564194 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.556543    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/becab9bd-103e-4d46-8e6c-fe8c86602dc5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb\" (UID: \"becab9bd-103e-4d46-8e6c-fe8c86602dc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"
Oct 30 19:04:32.564194 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.557432    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/becab9bd-103e-4d46-8e6c-fe8c86602dc5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb\" (UID: \"becab9bd-103e-4d46-8e6c-fe8c86602dc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"
Oct 30 19:04:32.564194 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.559190    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4sd5\" (UniqueName: \"kubernetes.io/projected/0fa40c2a-bd44-4da8-95b0-3f68cfd29335-kube-api-access-t4sd5\") pod \"payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8\" (UID: \"0fa40c2a-bd44-4da8-95b0-3f68cfd29335\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"
Oct 30 19:04:32.571850 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.571820    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-786kb\" (UniqueName: \"kubernetes.io/projected/becab9bd-103e-4d46-8e6c-fe8c86602dc5-kube-api-access-786kb\") pod \"payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb\" (UID: \"becab9bd-103e-4d46-8e6c-fe8c86602dc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"
Oct 30 19:04:32.746945 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.746862    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"
Oct 30 19:04:32.810407 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:32.807652    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"
Oct 30 19:04:33.038430 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:04:33.038396    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa40c2a_bd44_4da8_95b0_3f68cfd29335.slice/crio-ecb68ebb0ec142a135c6cad764684fa8fbee8ab5ac82aecf19d36be325b747b7 WatchSource:0}: Error finding container ecb68ebb0ec142a135c6cad764684fa8fbee8ab5ac82aecf19d36be325b747b7: Status 404 returned error can't find the container with id ecb68ebb0ec142a135c6cad764684fa8fbee8ab5ac82aecf19d36be325b747b7
Oct 30 19:04:33.038535 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:33.038419    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"]
Oct 30 19:04:33.106922 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:04:33.106888    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecab9bd_103e_4d46_8e6c_fe8c86602dc5.slice/crio-7e003da04233b370c0a0c883791b187b83e4ab4b2b8d2915518036fdc0aea588 WatchSource:0}: Error finding container 7e003da04233b370c0a0c883791b187b83e4ab4b2b8d2915518036fdc0aea588: Status 404 returned error can't find the container with id 7e003da04233b370c0a0c883791b187b83e4ab4b2b8d2915518036fdc0aea588
Oct 30 19:04:33.107912 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:33.107884    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"]
Oct 30 19:04:34.024179 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.024144    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb" event={"ID":"becab9bd-103e-4d46-8e6c-fe8c86602dc5","Type":"ContainerStarted","Data":"998924f08ed36e7eb0d9bfc9f71b3d827982c74a704951ad556ad2ebb7bf01ad"}
Oct 30 19:04:34.024698 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.024203    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb" event={"ID":"becab9bd-103e-4d46-8e6c-fe8c86602dc5","Type":"ContainerStarted","Data":"7e003da04233b370c0a0c883791b187b83e4ab4b2b8d2915518036fdc0aea588"}
Oct 30 19:04:34.024698 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.024231    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"
Oct 30 19:04:34.026064 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.026036    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8" event={"ID":"0fa40c2a-bd44-4da8-95b0-3f68cfd29335","Type":"ContainerStarted","Data":"079a434103b1b346a28cfc5a31482f12c374bf51316e9cffd404f8e356201f64"}
Oct 30 19:04:34.026216 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.026086    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8" event={"ID":"0fa40c2a-bd44-4da8-95b0-3f68cfd29335","Type":"ContainerStarted","Data":"ecb68ebb0ec142a135c6cad764684fa8fbee8ab5ac82aecf19d36be325b747b7"}
Oct 30 19:04:34.026216 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.026115    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"
Oct 30 19:04:34.073468 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.071863    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb" podStartSLOduration=2.071840758 podStartE2EDuration="2.071840758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:34.046727284 +0000 UTC m=+1887.915145614" watchObservedRunningTime="2026-10-30 19:04:34.071840758 +0000 UTC m=+1887.940259088"
Oct 30 19:04:34.114153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.114043    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8" podStartSLOduration=2.113992599 podStartE2EDuration="2.113992599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:34.080566287 +0000 UTC m=+1887.948984619" watchObservedRunningTime="2026-10-30 19:04:34.113992599 +0000 UTC m=+1887.982410930"
Oct 30 19:04:34.602167 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.602089    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 19:04:34.713411 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.713382    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-workload-socket\") pod \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") "
Oct 30 19:04:34.713590 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.713509    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/331f4d8f-f859-4c40-a26f-3e6ef07d840a-kube-api-access-lsf46\" (UniqueName: \"kubernetes.io/projected/331f4d8f-f859-4c40-a26f-3e6ef07d840a-kube-api-access-lsf46\") pod \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") "
Oct 30 19:04:34.713590 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.713566    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-podinfo\") pod \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") "
Oct 30 19:04:34.713590 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.713588    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-credential-socket\") pod \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") "
Oct 30 19:04:34.713771 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.713647    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-envoy\") pod \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") "
Oct 30 19:04:34.713771 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.713657    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-workload-socket" pod "331f4d8f-f859-4c40-a26f-3e6ef07d840a" (UID: "331f4d8f-f859-4c40-a26f-3e6ef07d840a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:34.713771 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.713669    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-workload-certs\") pod \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") "
Oct 30 19:04:34.713771 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.713720    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-data\") pod \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") "
Oct 30 19:04:34.714003 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.713795    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istiod-ca-cert\") pod \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") "
Oct 30 19:04:34.714003 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.713858    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-token\" (UniqueName: \"kubernetes.io/projected/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-token\") pod \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\" (UID: \"331f4d8f-f859-4c40-a26f-3e6ef07d840a\") "
Oct 30 19:04:34.714003 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.713889    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-credential-socket" pod "331f4d8f-f859-4c40-a26f-3e6ef07d840a" (UID: "331f4d8f-f859-4c40-a26f-3e6ef07d840a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:34.714188 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.714065    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-workload-certs" pod "331f4d8f-f859-4c40-a26f-3e6ef07d840a" (UID: "331f4d8f-f859-4c40-a26f-3e6ef07d840a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:34.714310 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.714266    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-data" pod "331f4d8f-f859-4c40-a26f-3e6ef07d840a" (UID: "331f4d8f-f859-4c40-a26f-3e6ef07d840a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:34.714423 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.714398    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istiod-ca-cert" pod "331f4d8f-f859-4c40-a26f-3e6ef07d840a" (UID: "331f4d8f-f859-4c40-a26f-3e6ef07d840a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:34.714656 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.714628    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:34.714656 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.714654    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:34.714804 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.714669    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:34.714804 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.714682    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:34.714804 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.714696    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:34.716379 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.716348    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-envoy" pod "331f4d8f-f859-4c40-a26f-3e6ef07d840a" (UID: "331f4d8f-f859-4c40-a26f-3e6ef07d840a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:34.716799 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.716765    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331f4d8f-f859-4c40-a26f-3e6ef07d840a-kube-api-access-lsf46" pod "331f4d8f-f859-4c40-a26f-3e6ef07d840a" (UID: "331f4d8f-f859-4c40-a26f-3e6ef07d840a"). InnerVolumeSpecName "kube-api-access-lsf46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:34.716799 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.716788    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-token" pod "331f4d8f-f859-4c40-a26f-3e6ef07d840a" (UID: "331f4d8f-f859-4c40-a26f-3e6ef07d840a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:34.716990 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.716795    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-podinfo" pod "331f4d8f-f859-4c40-a26f-3e6ef07d840a" (UID: "331f4d8f-f859-4c40-a26f-3e6ef07d840a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:04:34.815891 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.815859    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:34.815891 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.815889    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsf46\" (UniqueName: \"kubernetes.io/projected/331f4d8f-f859-4c40-a26f-3e6ef07d840a-kube-api-access-lsf46\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:34.815891 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.815899    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:34.816147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:34.815909    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331f4d8f-f859-4c40-a26f-3e6ef07d840a-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:35.035782 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:35.035694    4372 generic.go:356] "Generic (PLEG): container finished" podID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" containerID="6627b03ea5bee45d11b62146f879ba00305e251de7a63603474c31bec7b886cf" exitCode=0
Oct 30 19:04:35.035782 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:35.035740    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"
Oct 30 19:04:35.036356 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:35.035783    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" event={"ID":"331f4d8f-f859-4c40-a26f-3e6ef07d840a","Type":"ContainerDied","Data":"6627b03ea5bee45d11b62146f879ba00305e251de7a63603474c31bec7b886cf"}
Oct 30 19:04:35.036356 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:35.035844    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5" event={"ID":"331f4d8f-f859-4c40-a26f-3e6ef07d840a","Type":"ContainerDied","Data":"3a72f3480fcd74a315a1c5a2ed775a9e488e900845c31ca79f8605bdc2915dff"}
Oct 30 19:04:35.036356 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:35.035880    4372 scope.go:122] "RemoveContainer" containerID="6627b03ea5bee45d11b62146f879ba00305e251de7a63603474c31bec7b886cf"
Oct 30 19:04:35.047831 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:35.047812    4372 scope.go:122] "RemoveContainer" containerID="6627b03ea5bee45d11b62146f879ba00305e251de7a63603474c31bec7b886cf"
Oct 30 19:04:35.048102 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:04:35.048079    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6627b03ea5bee45d11b62146f879ba00305e251de7a63603474c31bec7b886cf\": container with ID starting with 6627b03ea5bee45d11b62146f879ba00305e251de7a63603474c31bec7b886cf not found: ID does not exist" containerID="6627b03ea5bee45d11b62146f879ba00305e251de7a63603474c31bec7b886cf"
Oct 30 19:04:35.048256 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:35.048120    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6627b03ea5bee45d11b62146f879ba00305e251de7a63603474c31bec7b886cf"} err="failed to get container status \"6627b03ea5bee45d11b62146f879ba00305e251de7a63603474c31bec7b886cf\": rpc error: code = NotFound desc = could not find container \"6627b03ea5bee45d11b62146f879ba00305e251de7a63603474c31bec7b886cf\": container with ID starting with 6627b03ea5bee45d11b62146f879ba00305e251de7a63603474c31bec7b886cf not found: ID does not exist"
Oct 30 19:04:35.066769 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:35.066746    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"]
Oct 30 19:04:35.070305 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:35.070287    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-sz6c5"]
Oct 30 19:04:36.267297 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:36.267268    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" path="/var/lib/kubelet/pods/331f4d8f-f859-4c40-a26f-3e6ef07d840a/volumes"
Oct 30 19:04:37.965345 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:37.965311    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"
Oct 30 19:04:37.965889 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:37.965428    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"
Oct 30 19:04:40.540494 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.540462    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"]
Oct 30 19:04:40.541195 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.541167    4372 cpu_manager.go:414] "containerMap: removing container" podUID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" containerName="istio-proxy"
Oct 30 19:04:40.541195 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.541185    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" containerName="istio-proxy"
Oct 30 19:04:40.541414 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.541329    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="331f4d8f-f859-4c40-a26f-3e6ef07d840a" containerName="istio-proxy"
Oct 30 19:04:40.546578 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.546554    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:40.567639 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.567612    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-21d896-serving-cert\""
Oct 30 19:04:40.572588 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.572566    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"]
Oct 30 19:04:40.650373 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.650331    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37daf29d-e5e7-49d4-a763-4b948871431f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:40.650630 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.650607    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37daf29d-e5e7-49d4-a763-4b948871431f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:40.650687 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.650673    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37daf29d-e5e7-49d4-a763-4b948871431f-metrics-certs\") pod \"maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:40.650784 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.650764    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhbz9\" (UniqueName: \"kubernetes.io/projected/37daf29d-e5e7-49d4-a763-4b948871431f-kube-api-access-lhbz9\") pod \"maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:40.753442 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.753397    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37daf29d-e5e7-49d4-a763-4b948871431f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:40.754427 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.754398    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37daf29d-e5e7-49d4-a763-4b948871431f-metrics-certs\") pod \"maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:40.788479 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.754661    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhbz9\" (UniqueName: \"kubernetes.io/projected/37daf29d-e5e7-49d4-a763-4b948871431f-kube-api-access-lhbz9\") pod \"maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:40.788479 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.783650    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37daf29d-e5e7-49d4-a763-4b948871431f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:40.788479 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.770933    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37daf29d-e5e7-49d4-a763-4b948871431f-metrics-certs\") pod \"maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:40.788479 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.781142    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhbz9\" (UniqueName: \"kubernetes.io/projected/37daf29d-e5e7-49d4-a763-4b948871431f-kube-api-access-lhbz9\") pod \"maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:40.788479 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.767419    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37daf29d-e5e7-49d4-a763-4b948871431f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:40.788479 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.784470    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37daf29d-e5e7-49d4-a763-4b948871431f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:40.884320 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:40.884290    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:41.267195 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:41.267073    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"]
Oct 30 19:04:41.286245 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:04:41.286214    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37daf29d_e5e7_49d4_a763_4b948871431f.slice/crio-3f22a036a64134c3f40f625ec9dd647f7e3e7860eb256400d0e53150813f33bf WatchSource:0}: Error finding container 3f22a036a64134c3f40f625ec9dd647f7e3e7860eb256400d0e53150813f33bf: Status 404 returned error can't find the container with id 3f22a036a64134c3f40f625ec9dd647f7e3e7860eb256400d0e53150813f33bf
Oct 30 19:04:41.288433 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:41.288412    4372 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:04:42.104301 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:42.104268    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv" event={"ID":"37daf29d-e5e7-49d4-a763-4b948871431f","Type":"ContainerStarted","Data":"f572f3b60656b6f6bb8d9c71d7274ab88e50f6778969af83cf78123268a28013"}
Oct 30 19:04:42.104802 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:42.104321    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv" event={"ID":"37daf29d-e5e7-49d4-a763-4b948871431f","Type":"ContainerStarted","Data":"3f22a036a64134c3f40f625ec9dd647f7e3e7860eb256400d0e53150813f33bf"}
Oct 30 19:04:42.104802 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:42.104350    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:42.127070 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:42.127020    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv" podStartSLOduration=1.664802856 podStartE2EDuration="2.127007122s" totalImagesPullingTime="462.204266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:42.125089254 +0000 UTC m=+1895.993507584" watchObservedRunningTime="2026-10-30 19:04:42.127007122 +0000 UTC m=+1895.995425451"
Oct 30 19:04:44.807036 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:44.805874    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"]
Oct 30 19:04:44.947052 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:44.945481    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"]
Oct 30 19:04:44.947052 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:44.945632    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:44.990532 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:44.990501    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e8c554-openshift-default-dockercfg-m67k9\""
Oct 30 19:04:45.038533 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.038502    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"
Oct 30 19:04:45.038805 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.038754    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"
Oct 30 19:04:45.109095 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.109060    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-data\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.109341 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.109166    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-workload-socket\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.109341 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.109229    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-envoy\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.109341 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.109282    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-workload-certs\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.109341 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.109315    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85hht\" (UniqueName: \"kubernetes.io/projected/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-kube-api-access-85hht\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.109583 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.109535    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istiod-ca-cert\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.109673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.109646    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-credential-socket\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.109731 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.109708    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-podinfo\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.109786 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.109751    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-token\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.210715 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.210684    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istiod-ca-cert\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.210885 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.210738    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-credential-socket\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.210885 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.210849    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-podinfo\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.210994 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.210897    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-token\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.211053 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.211039    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-data\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.211109 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.211072    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-workload-socket\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.211188 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.211040    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-credential-socket\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.211253 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.211199    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-envoy\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.211590 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.211560    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-workload-certs\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.211590 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.211590    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85hht\" (UniqueName: \"kubernetes.io/projected/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-kube-api-access-85hht\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.211931 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.211482    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istiod-ca-cert\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.211931 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.211310    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-data\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.211931 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.211896    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-workload-certs\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.211931 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.211413    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-workload-socket\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.213537 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.213514    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-envoy\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.213636 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.213578    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-podinfo\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.223666 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.223637    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-token\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.224374 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.224354    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85hht\" (UniqueName: \"kubernetes.io/projected/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-kube-api-access-85hht\") pod \"e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") " pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.300164 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.295587    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:45.508602 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.508563    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"]
Oct 30 19:04:45.704203 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.702182    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"]
Oct 30 19:04:45.812591 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.812543    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:04:45.812951 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.812647    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:04:45.812951 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:45.812693    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:04:46.136621 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:46.136582    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" event={"ID":"a6b96f6e-b871-499b-ab8f-18fffaa3ece2","Type":"ContainerStarted","Data":"c1d6b783d5f6c51bb463972000e9bc0b29fa67bd71004c563bde6a647dcbe247"}
Oct 30 19:04:46.136810 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:46.136652    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" event={"ID":"a6b96f6e-b871-499b-ab8f-18fffaa3ece2","Type":"ContainerStarted","Data":"cf39e7a110ac0c7d391b2db897f3fc5279c6b9fa21f32ca669f37c7f7abc2720"}
Oct 30 19:04:46.184701 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:46.184653    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" podStartSLOduration=2.184639435 podStartE2EDuration="2.184639435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:46.182329443 +0000 UTC m=+1900.050747774" watchObservedRunningTime="2026-10-30 19:04:46.184639435 +0000 UTC m=+1900.053057764"
Oct 30 19:04:46.296951 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:46.296921    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:46.303213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:46.303189    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:47.143828 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:47.143794    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:47.144626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:47.144606    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:04:48.118381 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:48.118356    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:04:57.696280 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:04:57.696244    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:04:57.935948 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:04:57.935915    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"]
Oct 30 19:05:00.535637 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.535589    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw" podUID="4fcba8ac-337e-4ee9-9828-a9869f075500" containerName="main" containerID="cri-o://90bd040460fcc342365cc1523899bca09ad9b581541e8b905d728549c03f3f99" gracePeriod=45
Oct 30 19:05:00.830887 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.830858    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:05:00.857544 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.857517    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fcba8ac-337e-4ee9-9828-a9869f075500-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcba8ac-337e-4ee9-9828-a9869f075500-tls-certs\") pod \"4fcba8ac-337e-4ee9-9828-a9869f075500\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") "
Oct 30 19:05:00.857702 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.857556    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fcba8ac-337e-4ee9-9828-a9869f075500-kube-api-access-kp9vc\" (UniqueName: \"kubernetes.io/projected/4fcba8ac-337e-4ee9-9828-a9869f075500-kube-api-access-kp9vc\") pod \"4fcba8ac-337e-4ee9-9828-a9869f075500\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") "
Oct 30 19:05:00.857702 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.857583    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-home\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-home\") pod \"4fcba8ac-337e-4ee9-9828-a9869f075500\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") "
Oct 30 19:05:00.857871 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.857717    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-model-cache\") pod \"4fcba8ac-337e-4ee9-9828-a9869f075500\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") "
Oct 30 19:05:00.857871 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.857851    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-home" pod "4fcba8ac-337e-4ee9-9828-a9869f075500" (UID: "4fcba8ac-337e-4ee9-9828-a9869f075500"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:00.857998 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.857968    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-model-cache" pod "4fcba8ac-337e-4ee9-9828-a9869f075500" (UID: "4fcba8ac-337e-4ee9-9828-a9869f075500"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:00.858059 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.858040    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-dshm\") pod \"4fcba8ac-337e-4ee9-9828-a9869f075500\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") "
Oct 30 19:05:00.858112 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.858083    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-tmp-dir\") pod \"4fcba8ac-337e-4ee9-9828-a9869f075500\" (UID: \"4fcba8ac-337e-4ee9-9828-a9869f075500\") "
Oct 30 19:05:00.858373 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.858347    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-tmp-dir" pod "4fcba8ac-337e-4ee9-9828-a9869f075500" (UID: "4fcba8ac-337e-4ee9-9828-a9869f075500"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:00.858733 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.858705    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:00.858809 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.858734    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:00.858809 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.858744    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:00.860540 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.860515    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fcba8ac-337e-4ee9-9828-a9869f075500-kube-api-access-kp9vc" pod "4fcba8ac-337e-4ee9-9828-a9869f075500" (UID: "4fcba8ac-337e-4ee9-9828-a9869f075500"). InnerVolumeSpecName "kube-api-access-kp9vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:00.860694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.860669    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fcba8ac-337e-4ee9-9828-a9869f075500-tls-certs" pod "4fcba8ac-337e-4ee9-9828-a9869f075500" (UID: "4fcba8ac-337e-4ee9-9828-a9869f075500"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:00.860694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.860678    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-dshm" pod "4fcba8ac-337e-4ee9-9828-a9869f075500" (UID: "4fcba8ac-337e-4ee9-9828-a9869f075500"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:00.959923 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.959885    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcba8ac-337e-4ee9-9828-a9869f075500-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:00.959923 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.959917    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcba8ac-337e-4ee9-9828-a9869f075500-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:00.960111 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:00.959954    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp9vc\" (UniqueName: \"kubernetes.io/projected/4fcba8ac-337e-4ee9-9828-a9869f075500-kube-api-access-kp9vc\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:01.423527 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:01.423496    4372 generic.go:356] "Generic (PLEG): container finished" podID="4fcba8ac-337e-4ee9-9828-a9869f075500" containerID="90bd040460fcc342365cc1523899bca09ad9b581541e8b905d728549c03f3f99" exitCode=0
Oct 30 19:05:01.423703 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:01.423552    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"
Oct 30 19:05:01.423703 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:01.423586    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw" event={"ID":"4fcba8ac-337e-4ee9-9828-a9869f075500","Type":"ContainerDied","Data":"90bd040460fcc342365cc1523899bca09ad9b581541e8b905d728549c03f3f99"}
Oct 30 19:05:01.423703 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:01.423635    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw" event={"ID":"4fcba8ac-337e-4ee9-9828-a9869f075500","Type":"ContainerDied","Data":"8457c22d26402c4c2b151f91fd34eaac73a872ff1e25f9d5828dfad1dbd7fc4d"}
Oct 30 19:05:01.423703 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:01.423654    4372 scope.go:122] "RemoveContainer" containerID="90bd040460fcc342365cc1523899bca09ad9b581541e8b905d728549c03f3f99"
Oct 30 19:05:01.434748 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:01.434733    4372 scope.go:122] "RemoveContainer" containerID="90bd040460fcc342365cc1523899bca09ad9b581541e8b905d728549c03f3f99"
Oct 30 19:05:01.435020 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:05:01.435000    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90bd040460fcc342365cc1523899bca09ad9b581541e8b905d728549c03f3f99\": container with ID starting with 90bd040460fcc342365cc1523899bca09ad9b581541e8b905d728549c03f3f99 not found: ID does not exist" containerID="90bd040460fcc342365cc1523899bca09ad9b581541e8b905d728549c03f3f99"
Oct 30 19:05:01.435090 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:01.435029    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90bd040460fcc342365cc1523899bca09ad9b581541e8b905d728549c03f3f99"} err="failed to get container status \"90bd040460fcc342365cc1523899bca09ad9b581541e8b905d728549c03f3f99\": rpc error: code = NotFound desc = could not find container \"90bd040460fcc342365cc1523899bca09ad9b581541e8b905d728549c03f3f99\": container with ID starting with 90bd040460fcc342365cc1523899bca09ad9b581541e8b905d728549c03f3f99 not found: ID does not exist"
Oct 30 19:05:01.450711 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:01.450689    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"]
Oct 30 19:05:01.462295 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:01.462274    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-304f22/auth-test-model-304f22-kserve-658758d7fc-gdhrw"]
Oct 30 19:05:01.950666 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:01.950635    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"]
Oct 30 19:05:01.951052 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:01.950893    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy" containerID="cri-o://58a14207ece4febcdef46fcbb504f6ced2a1370b81c4b7582bba42a8fc8197bb" gracePeriod=30
Oct 30 19:05:02.267385 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:02.267304    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fcba8ac-337e-4ee9-9828-a9869f075500" path="/var/lib/kubelet/pods/4fcba8ac-337e-4ee9-9828-a9869f075500/volumes"
Oct 30 19:05:07.235335 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.235279    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 19:05:07.391743 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.391709    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-workload-socket\") pod \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") "
Oct 30 19:05:07.391948 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.391819    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-workload-certs\") pod \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") "
Oct 30 19:05:07.392012 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.391967    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-workload-socket" pod "7ce78af1-d7d9-4019-ab03-aa84fda13e1c" (UID: "7ce78af1-d7d9-4019-ab03-aa84fda13e1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:07.392066 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.392021    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-envoy\") pod \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") "
Oct 30 19:05:07.392119 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.392103    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-workload-certs" pod "7ce78af1-d7d9-4019-ab03-aa84fda13e1c" (UID: "7ce78af1-d7d9-4019-ab03-aa84fda13e1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:07.392205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.392159    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istiod-ca-cert\") pod \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") "
Oct 30 19:05:07.392205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.392195    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-podinfo\") pod \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") "
Oct 30 19:05:07.392316 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.392240    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-kube-api-access-rdzjn\" (UniqueName: \"kubernetes.io/projected/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-kube-api-access-rdzjn\") pod \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") "
Oct 30 19:05:07.392316 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.392290    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-data\") pod \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") "
Oct 30 19:05:07.392418 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.392338    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-credential-socket\") pod \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") "
Oct 30 19:05:07.392418 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.392376    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-token\" (UniqueName: \"kubernetes.io/projected/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-token\") pod \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\" (UID: \"7ce78af1-d7d9-4019-ab03-aa84fda13e1c\") "
Oct 30 19:05:07.392635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.392602    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-data" pod "7ce78af1-d7d9-4019-ab03-aa84fda13e1c" (UID: "7ce78af1-d7d9-4019-ab03-aa84fda13e1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:07.392635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.392620    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istiod-ca-cert" pod "7ce78af1-d7d9-4019-ab03-aa84fda13e1c" (UID: "7ce78af1-d7d9-4019-ab03-aa84fda13e1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:07.392854 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.392639    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-credential-socket" pod "7ce78af1-d7d9-4019-ab03-aa84fda13e1c" (UID: "7ce78af1-d7d9-4019-ab03-aa84fda13e1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:07.393483 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.393418    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:07.393483 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.393448    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:07.393483 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.393464    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:07.393483 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.393480    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:07.393483 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.393494    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:07.394507 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.394486    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-envoy" pod "7ce78af1-d7d9-4019-ab03-aa84fda13e1c" (UID: "7ce78af1-d7d9-4019-ab03-aa84fda13e1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:07.394654 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.394635    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-podinfo" pod "7ce78af1-d7d9-4019-ab03-aa84fda13e1c" (UID: "7ce78af1-d7d9-4019-ab03-aa84fda13e1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:05:07.394706 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.394665    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-kube-api-access-rdzjn" pod "7ce78af1-d7d9-4019-ab03-aa84fda13e1c" (UID: "7ce78af1-d7d9-4019-ab03-aa84fda13e1c"). InnerVolumeSpecName "kube-api-access-rdzjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:07.395185 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.395170    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-token" pod "7ce78af1-d7d9-4019-ab03-aa84fda13e1c" (UID: "7ce78af1-d7d9-4019-ab03-aa84fda13e1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:07.468030 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.468000    4372 generic.go:356] "Generic (PLEG): container finished" podID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerID="58a14207ece4febcdef46fcbb504f6ced2a1370b81c4b7582bba42a8fc8197bb" exitCode=0
Oct 30 19:05:07.468163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.468058    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"
Oct 30 19:05:07.468163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.468082    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" event={"ID":"7ce78af1-d7d9-4019-ab03-aa84fda13e1c","Type":"ContainerDied","Data":"58a14207ece4febcdef46fcbb504f6ced2a1370b81c4b7582bba42a8fc8197bb"}
Oct 30 19:05:07.468245 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.468168    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk" event={"ID":"7ce78af1-d7d9-4019-ab03-aa84fda13e1c","Type":"ContainerDied","Data":"ec57f782c5db70b308a92709aebb7ca9fb69d955dee576ff7de98ffc444ecd5d"}
Oct 30 19:05:07.468245 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.468196    4372 scope.go:122] "RemoveContainer" containerID="58a14207ece4febcdef46fcbb504f6ced2a1370b81c4b7582bba42a8fc8197bb"
Oct 30 19:05:07.479393 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.479378    4372 scope.go:122] "RemoveContainer" containerID="58a14207ece4febcdef46fcbb504f6ced2a1370b81c4b7582bba42a8fc8197bb"
Oct 30 19:05:07.479648 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:05:07.479630    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58a14207ece4febcdef46fcbb504f6ced2a1370b81c4b7582bba42a8fc8197bb\": container with ID starting with 58a14207ece4febcdef46fcbb504f6ced2a1370b81c4b7582bba42a8fc8197bb not found: ID does not exist" containerID="58a14207ece4febcdef46fcbb504f6ced2a1370b81c4b7582bba42a8fc8197bb"
Oct 30 19:05:07.479725 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.479654    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58a14207ece4febcdef46fcbb504f6ced2a1370b81c4b7582bba42a8fc8197bb"} err="failed to get container status \"58a14207ece4febcdef46fcbb504f6ced2a1370b81c4b7582bba42a8fc8197bb\": rpc error: code = NotFound desc = could not find container \"58a14207ece4febcdef46fcbb504f6ced2a1370b81c4b7582bba42a8fc8197bb\": container with ID starting with 58a14207ece4febcdef46fcbb504f6ced2a1370b81c4b7582bba42a8fc8197bb not found: ID does not exist"
Oct 30 19:05:07.494954 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.494904    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:07.494954 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.494928    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:07.494954 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.494938    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:07.494954 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.494948    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdzjn\" (UniqueName: \"kubernetes.io/projected/7ce78af1-d7d9-4019-ab03-aa84fda13e1c-kube-api-access-rdzjn\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:07.508093 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.508070    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"]
Oct 30 19:05:07.511735 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:07.511715    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-pbvxk"]
Oct 30 19:05:08.266471 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:08.266441    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" path="/var/lib/kubelet/pods/7ce78af1-d7d9-4019-ab03-aa84fda13e1c/volumes"
Oct 30 19:05:12.575149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:12.573039    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"]
Oct 30 19:05:12.575149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:12.573354    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" podUID="450e8016-b275-4172-ae20-5668c854e43a" containerName="istio-proxy" containerID="cri-o://75ffe7ae3d272a243f1e65521b236a993a45f30dc4ed3ff5ac94417e214d988d" gracePeriod=30
Oct 30 19:05:12.962522 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:12.962412    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt" podUID="e1875d62-38cc-452b-87f0-5b78523a0d41" containerName="main" containerID="cri-o://3a2349e82c8b8653bb59ebf0e8ba370e551991bec3b567ac922f547610ff635d" gracePeriod=45
Oct 30 19:05:13.284090 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.284064    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:05:13.293923 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.293898    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-tmp-dir\") pod \"e1875d62-38cc-452b-87f0-5b78523a0d41\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") "
Oct 30 19:05:13.294069 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.294054    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1875d62-38cc-452b-87f0-5b78523a0d41-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1875d62-38cc-452b-87f0-5b78523a0d41-tls-certs\") pod \"e1875d62-38cc-452b-87f0-5b78523a0d41\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") "
Oct 30 19:05:13.294162 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.294119    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-tmp-dir" pod "e1875d62-38cc-452b-87f0-5b78523a0d41" (UID: "e1875d62-38cc-452b-87f0-5b78523a0d41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:13.294237 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.294190    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-dshm\") pod \"e1875d62-38cc-452b-87f0-5b78523a0d41\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") "
Oct 30 19:05:13.294303 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.294236    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-model-cache\") pod \"e1875d62-38cc-452b-87f0-5b78523a0d41\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") "
Oct 30 19:05:13.294303 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.294266    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1875d62-38cc-452b-87f0-5b78523a0d41-kube-api-access-nnvjr\" (UniqueName: \"kubernetes.io/projected/e1875d62-38cc-452b-87f0-5b78523a0d41-kube-api-access-nnvjr\") pod \"e1875d62-38cc-452b-87f0-5b78523a0d41\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") "
Oct 30 19:05:13.294438 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.294421    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-home\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-home\") pod \"e1875d62-38cc-452b-87f0-5b78523a0d41\" (UID: \"e1875d62-38cc-452b-87f0-5b78523a0d41\") "
Oct 30 19:05:13.294506 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.294488    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-model-cache" pod "e1875d62-38cc-452b-87f0-5b78523a0d41" (UID: "e1875d62-38cc-452b-87f0-5b78523a0d41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:13.294784 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.294759    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-home" pod "e1875d62-38cc-452b-87f0-5b78523a0d41" (UID: "e1875d62-38cc-452b-87f0-5b78523a0d41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:13.295340 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.295316    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:13.295434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.295346    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:13.295434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.295364    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:13.296668 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.296648    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-dshm" pod "e1875d62-38cc-452b-87f0-5b78523a0d41" (UID: "e1875d62-38cc-452b-87f0-5b78523a0d41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:13.296775 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.296757    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1875d62-38cc-452b-87f0-5b78523a0d41-tls-certs" pod "e1875d62-38cc-452b-87f0-5b78523a0d41" (UID: "e1875d62-38cc-452b-87f0-5b78523a0d41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:13.296838 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.296822    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1875d62-38cc-452b-87f0-5b78523a0d41-kube-api-access-nnvjr" pod "e1875d62-38cc-452b-87f0-5b78523a0d41" (UID: "e1875d62-38cc-452b-87f0-5b78523a0d41"). InnerVolumeSpecName "kube-api-access-nnvjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:13.396969 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.396938    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1875d62-38cc-452b-87f0-5b78523a0d41-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:13.396969 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.396963    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1875d62-38cc-452b-87f0-5b78523a0d41-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:13.396969 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.396973    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnvjr\" (UniqueName: \"kubernetes.io/projected/e1875d62-38cc-452b-87f0-5b78523a0d41-kube-api-access-nnvjr\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:13.771621 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.771587    4372 generic.go:356] "Generic (PLEG): container finished" podID="e1875d62-38cc-452b-87f0-5b78523a0d41" containerID="3a2349e82c8b8653bb59ebf0e8ba370e551991bec3b567ac922f547610ff635d" exitCode=0
Oct 30 19:05:13.772085 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.771640    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"
Oct 30 19:05:13.772085 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.771650    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt" event={"ID":"e1875d62-38cc-452b-87f0-5b78523a0d41","Type":"ContainerDied","Data":"3a2349e82c8b8653bb59ebf0e8ba370e551991bec3b567ac922f547610ff635d"}
Oct 30 19:05:13.772085 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.771682    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt" event={"ID":"e1875d62-38cc-452b-87f0-5b78523a0d41","Type":"ContainerDied","Data":"cb0863c6487939bc8ad168d03a3870d7991c4b22d9c86ef5295ddd97c0a0b340"}
Oct 30 19:05:13.772085 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.771697    4372 scope.go:122] "RemoveContainer" containerID="3a2349e82c8b8653bb59ebf0e8ba370e551991bec3b567ac922f547610ff635d"
Oct 30 19:05:13.784965 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.784948    4372 scope.go:122] "RemoveContainer" containerID="3a2349e82c8b8653bb59ebf0e8ba370e551991bec3b567ac922f547610ff635d"
Oct 30 19:05:13.785266 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:05:13.785238    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2349e82c8b8653bb59ebf0e8ba370e551991bec3b567ac922f547610ff635d\": container with ID starting with 3a2349e82c8b8653bb59ebf0e8ba370e551991bec3b567ac922f547610ff635d not found: ID does not exist" containerID="3a2349e82c8b8653bb59ebf0e8ba370e551991bec3b567ac922f547610ff635d"
Oct 30 19:05:13.785364 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.785273    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2349e82c8b8653bb59ebf0e8ba370e551991bec3b567ac922f547610ff635d"} err="failed to get container status \"3a2349e82c8b8653bb59ebf0e8ba370e551991bec3b567ac922f547610ff635d\": rpc error: code = NotFound desc = could not find container \"3a2349e82c8b8653bb59ebf0e8ba370e551991bec3b567ac922f547610ff635d\": container with ID starting with 3a2349e82c8b8653bb59ebf0e8ba370e551991bec3b567ac922f547610ff635d not found: ID does not exist"
Oct 30 19:05:13.799267 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.799238    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"]
Oct 30 19:05:13.802977 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:13.802953    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6d7870/auth-test-model-6d7870-kserve-54c956f97-wsrbt"]
Oct 30 19:05:14.267401 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:14.267371    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1875d62-38cc-452b-87f0-5b78523a0d41" path="/var/lib/kubelet/pods/e1875d62-38cc-452b-87f0-5b78523a0d41/volumes"
Oct 30 19:05:15.511863 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:15.511826    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:05:17.926564 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:17.926521    4372 generic.go:356] "Generic (PLEG): container finished" podID="450e8016-b275-4172-ae20-5668c854e43a" containerID="75ffe7ae3d272a243f1e65521b236a993a45f30dc4ed3ff5ac94417e214d988d" exitCode=0
Oct 30 19:05:17.927073 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:17.926642    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" event={"ID":"450e8016-b275-4172-ae20-5668c854e43a","Type":"ContainerDied","Data":"75ffe7ae3d272a243f1e65521b236a993a45f30dc4ed3ff5ac94417e214d988d"}
Oct 30 19:05:18.064567 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.064541    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 19:05:18.127397 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.127353    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-istio-envoy\") pod \"450e8016-b275-4172-ae20-5668c854e43a\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") "
Oct 30 19:05:18.127597 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.127499    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-workload-socket\") pod \"450e8016-b275-4172-ae20-5668c854e43a\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") "
Oct 30 19:05:18.127597 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.127553    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450e8016-b275-4172-ae20-5668c854e43a-istio-token\" (UniqueName: \"kubernetes.io/projected/450e8016-b275-4172-ae20-5668c854e43a-istio-token\") pod \"450e8016-b275-4172-ae20-5668c854e43a\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") "
Oct 30 19:05:18.127597 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.127580    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-istio-data\") pod \"450e8016-b275-4172-ae20-5668c854e43a\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") "
Oct 30 19:05:18.127784 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.127629    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450e8016-b275-4172-ae20-5668c854e43a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/450e8016-b275-4172-ae20-5668c854e43a-istiod-ca-cert\") pod \"450e8016-b275-4172-ae20-5668c854e43a\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") "
Oct 30 19:05:18.127784 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.127761    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-workload-certs\") pod \"450e8016-b275-4172-ae20-5668c854e43a\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") "
Oct 30 19:05:18.128254 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.128175    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450e8016-b275-4172-ae20-5668c854e43a-kube-api-access-hk86l\" (UniqueName: \"kubernetes.io/projected/450e8016-b275-4172-ae20-5668c854e43a-kube-api-access-hk86l\") pod \"450e8016-b275-4172-ae20-5668c854e43a\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") "
Oct 30 19:05:18.128254 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.128219    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-credential-socket\") pod \"450e8016-b275-4172-ae20-5668c854e43a\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") "
Oct 30 19:05:18.128254 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.128205    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-workload-socket" pod "450e8016-b275-4172-ae20-5668c854e43a" (UID: "450e8016-b275-4172-ae20-5668c854e43a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.128496 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.128346    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/450e8016-b275-4172-ae20-5668c854e43a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/450e8016-b275-4172-ae20-5668c854e43a-istio-podinfo\") pod \"450e8016-b275-4172-ae20-5668c854e43a\" (UID: \"450e8016-b275-4172-ae20-5668c854e43a\") "
Oct 30 19:05:18.128496 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.128368    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450e8016-b275-4172-ae20-5668c854e43a-istiod-ca-cert" pod "450e8016-b275-4172-ae20-5668c854e43a" (UID: "450e8016-b275-4172-ae20-5668c854e43a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:18.128958 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.128669    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-workload-certs" pod "450e8016-b275-4172-ae20-5668c854e43a" (UID: "450e8016-b275-4172-ae20-5668c854e43a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.128958 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.128934    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-credential-socket" pod "450e8016-b275-4172-ae20-5668c854e43a" (UID: "450e8016-b275-4172-ae20-5668c854e43a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.129192 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.129166    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-istio-data" pod "450e8016-b275-4172-ae20-5668c854e43a" (UID: "450e8016-b275-4172-ae20-5668c854e43a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.129677 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.129654    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.129773 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.129688    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.129773 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.129705    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.129773 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.129730    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.129773 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.129744    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/450e8016-b275-4172-ae20-5668c854e43a-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.130857 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.130823    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450e8016-b275-4172-ae20-5668c854e43a-kube-api-access-hk86l" pod "450e8016-b275-4172-ae20-5668c854e43a" (UID: "450e8016-b275-4172-ae20-5668c854e43a"). InnerVolumeSpecName "kube-api-access-hk86l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:18.133194 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.131806    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-istio-envoy" pod "450e8016-b275-4172-ae20-5668c854e43a" (UID: "450e8016-b275-4172-ae20-5668c854e43a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.133194 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.131935    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/450e8016-b275-4172-ae20-5668c854e43a-istio-podinfo" pod "450e8016-b275-4172-ae20-5668c854e43a" (UID: "450e8016-b275-4172-ae20-5668c854e43a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:05:18.133194 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.132860    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450e8016-b275-4172-ae20-5668c854e43a-istio-token" pod "450e8016-b275-4172-ae20-5668c854e43a" (UID: "450e8016-b275-4172-ae20-5668c854e43a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:18.240142 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.240083    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/450e8016-b275-4172-ae20-5668c854e43a-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.240142 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.240147    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/450e8016-b275-4172-ae20-5668c854e43a-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.240348 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.240163    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/450e8016-b275-4172-ae20-5668c854e43a-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.240348 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.240176    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk86l\" (UniqueName: \"kubernetes.io/projected/450e8016-b275-4172-ae20-5668c854e43a-kube-api-access-hk86l\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.974148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.968104    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw" event={"ID":"450e8016-b275-4172-ae20-5668c854e43a","Type":"ContainerDied","Data":"d74ced4bee3dc6657ef2a5ea50870f888f66053515993e1371817ebc33825d70"}
Oct 30 19:05:18.974148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.972227    4372 scope.go:122] "RemoveContainer" containerID="75ffe7ae3d272a243f1e65521b236a993a45f30dc4ed3ff5ac94417e214d988d"
Oct 30 19:05:18.974148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:18.972430    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"
Oct 30 19:05:19.059152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:19.054321    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"]
Oct 30 19:05:19.069149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:19.066492    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-ltppw"]
Oct 30 19:05:20.266903 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.266871    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450e8016-b275-4172-ae20-5668c854e43a" path="/var/lib/kubelet/pods/450e8016-b275-4172-ae20-5668c854e43a/volumes"
Oct 30 19:05:20.303215 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.303178    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"]
Oct 30 19:05:20.303944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.303922    4372 cpu_manager.go:414] "containerMap: removing container" podUID="450e8016-b275-4172-ae20-5668c854e43a" containerName="istio-proxy"
Oct 30 19:05:20.304056 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.303949    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450e8016-b275-4172-ae20-5668c854e43a" containerName="istio-proxy"
Oct 30 19:05:20.304056 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.303983    4372 cpu_manager.go:414] "containerMap: removing container" podUID="4fcba8ac-337e-4ee9-9828-a9869f075500" containerName="main"
Oct 30 19:05:20.304056 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.303992    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fcba8ac-337e-4ee9-9828-a9869f075500" containerName="main"
Oct 30 19:05:20.304056 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.304010    4372 cpu_manager.go:414] "containerMap: removing container" podUID="e1875d62-38cc-452b-87f0-5b78523a0d41" containerName="main"
Oct 30 19:05:20.304056 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.304017    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1875d62-38cc-452b-87f0-5b78523a0d41" containerName="main"
Oct 30 19:05:20.304056 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.304039    4372 cpu_manager.go:414] "containerMap: removing container" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy"
Oct 30 19:05:20.304056 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.304046    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy"
Oct 30 19:05:20.304466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.304234    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="450e8016-b275-4172-ae20-5668c854e43a" containerName="istio-proxy"
Oct 30 19:05:20.304466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.304256    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fcba8ac-337e-4ee9-9828-a9869f075500" containerName="main"
Oct 30 19:05:20.304466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.304268    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1875d62-38cc-452b-87f0-5b78523a0d41" containerName="main"
Oct 30 19:05:20.304466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.304282    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ce78af1-d7d9-4019-ab03-aa84fda13e1c" containerName="istio-proxy"
Oct 30 19:05:20.309733 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.309696    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"]
Oct 30 19:05:20.309874 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.309811    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:20.314742 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.314713    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"payload-processing-external-model-e2e--342ea960-dockercfg-f6pzj\""
Oct 30 19:05:20.314892 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.314774    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"agc-cred-5156225dde7c\""
Oct 30 19:05:20.315427 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.315403    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f7c6e3\""
Oct 30 19:05:20.315642 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.315623    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"routing-overlay\""
Oct 30 19:05:20.315748 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.315671    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"kube-root-ca.crt\""
Oct 30 19:05:20.315748 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.315711    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"openshift-service-ca.crt\""
Oct 30 19:05:20.331947 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.331920    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de7e1b40-890b-4297-b605-e948a6170c7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:20.332064 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.331964    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de7e1b40-890b-4297-b605-e948a6170c7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:20.332223 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.332202    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mrmq\" (UniqueName: \"kubernetes.io/projected/de7e1b40-890b-4297-b605-e948a6170c7e-kube-api-access-5mrmq\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:20.332300 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.332239    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de7e1b40-890b-4297-b605-e948a6170c7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:20.462185 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.443162    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de7e1b40-890b-4297-b605-e948a6170c7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:20.462185 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.443227    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de7e1b40-890b-4297-b605-e948a6170c7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:20.462185 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.443412    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mrmq\" (UniqueName: \"kubernetes.io/projected/de7e1b40-890b-4297-b605-e948a6170c7e-kube-api-access-5mrmq\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:20.462185 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.443446    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de7e1b40-890b-4297-b605-e948a6170c7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:20.462185 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.448298    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de7e1b40-890b-4297-b605-e948a6170c7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:20.462185 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.449372    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de7e1b40-890b-4297-b605-e948a6170c7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:20.462185 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.456835    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de7e1b40-890b-4297-b605-e948a6170c7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:20.493155 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.483684    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mrmq\" (UniqueName: \"kubernetes.io/projected/de7e1b40-890b-4297-b605-e948a6170c7e-kube-api-access-5mrmq\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:20.630444 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:20.630407    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:21.009262 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:05:21.009226    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7e1b40_890b_4297_b605_e948a6170c7e.slice/crio-4b534ad9fa3d0dcc87e3942aad25a0e71c3dacc31f51f00179440e8768f146dc WatchSource:0}: Error finding container 4b534ad9fa3d0dcc87e3942aad25a0e71c3dacc31f51f00179440e8768f146dc: Status 404 returned error can't find the container with id 4b534ad9fa3d0dcc87e3942aad25a0e71c3dacc31f51f00179440e8768f146dc
Oct 30 19:05:21.009694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:21.009666    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"]
Oct 30 19:05:22.019380 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:22.019343    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk" event={"ID":"de7e1b40-890b-4297-b605-e948a6170c7e","Type":"ContainerStarted","Data":"d54edf8ac261a51c0d7a9e0a2a25f3bdb71f46e8a0346bbb48ec766d138ea9a0"}
Oct 30 19:05:22.019911 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:22.019498    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:22.019911 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:22.019543    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk" event={"ID":"de7e1b40-890b-4297-b605-e948a6170c7e","Type":"ContainerStarted","Data":"4b534ad9fa3d0dcc87e3942aad25a0e71c3dacc31f51f00179440e8768f146dc"}
Oct 30 19:05:22.045836 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:22.045790    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk" podStartSLOduration=2.045777246 podStartE2EDuration="2.045777246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:22.044157951 +0000 UTC m=+1935.912576272" watchObservedRunningTime="2026-10-30 19:05:22.045777246 +0000 UTC m=+1935.914195577"
Oct 30 19:05:27.696585 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:05:27.696547    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:05:28.072434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:28.072396    4372 generic.go:356] "Generic (PLEG): container finished" podID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerID="ca4ddc6ed25aa2e36a7bb17470f3deea14e1f50bde13517ccbfd8a2b096691ee" exitCode=0
Oct 30 19:05:28.072613 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:28.072465    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k" event={"ID":"8b6ca59e-1c18-468a-8ba1-23aca6351823","Type":"ContainerDied","Data":"ca4ddc6ed25aa2e36a7bb17470f3deea14e1f50bde13517ccbfd8a2b096691ee"}
Oct 30 19:05:28.073000 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:28.072984    4372 scope.go:122] "RemoveContainer" containerID="ca4ddc6ed25aa2e36a7bb17470f3deea14e1f50bde13517ccbfd8a2b096691ee"
Oct 30 19:05:28.074585 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:28.074553    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:05:28.074677 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:28.074625    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:05:28.074677 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:28.074656    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:05:29.121152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:29.120199    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k" event={"ID":"8b6ca59e-1c18-468a-8ba1-23aca6351823","Type":"ContainerStarted","Data":"088cb1a291043e1ae52f28d86bbe4e63b14a95d3de0615968256942148e13c19"}
Oct 30 19:05:29.169295 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:29.169269    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:05:29.169459 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:29.169424    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:05:29.169504 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:29.169467    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k" podUID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 19:05:29.511835 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:29.511723    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"]
Oct 30 19:05:29.512027 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:29.511961    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" podUID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" containerName="istio-proxy" containerID="cri-o://8a096cb4590e3de82bd3c9185343f0edc131a2c8a682eaf9268da1d60cc53838" gracePeriod=30
Oct 30 19:05:30.169296 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:30.169258    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:05:30.169758 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:30.169316    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k" podUID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 19:05:31.169391 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:31.169361    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:05:31.169834 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:31.169410    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k" podUID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 19:05:32.008488 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:32.008389    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"]
Oct 30 19:05:32.008698 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:32.008586    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54" podUID="f5de6051-31b1-4529-8db0-d6cb1320a743" containerName="istio-proxy" containerID="cri-o://dbe1624a94eafa2e8363f87d6ef62d97bb15e1be8cfa1eaa7c113031f6d9e674" gracePeriod=30
Oct 30 19:05:32.169783 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:32.169748    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:05:32.170218 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:32.169807    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k" podUID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 19:05:33.030873 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:33.030827    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:05:33.169294 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:33.169259    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:05:33.169466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:33.169324    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k" podUID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 19:05:34.171075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.170678    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:05:34.171075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.170758    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k" podUID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 19:05:34.813818 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.813790    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 19:05:34.858719 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.858686    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-token\" (UniqueName: \"kubernetes.io/projected/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-token\") pod \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") "
Oct 30 19:05:34.858927 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.858746    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istiod-ca-cert\") pod \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") "
Oct 30 19:05:34.858927 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.858774    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-envoy\") pod \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") "
Oct 30 19:05:34.858927 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.858877    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-credential-socket\") pod \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") "
Oct 30 19:05:34.859154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.858954    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-workload-socket\") pod \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") "
Oct 30 19:05:34.859154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.859033    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-data\") pod \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") "
Oct 30 19:05:34.859282 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.859191    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-kube-api-access-cnkf6\" (UniqueName: \"kubernetes.io/projected/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-kube-api-access-cnkf6\") pod \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") "
Oct 30 19:05:34.859282 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.859236    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istiod-ca-cert" pod "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" (UID: "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:34.859282 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.859269    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-workload-certs\") pod \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") "
Oct 30 19:05:34.859449 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.859329    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-podinfo\") pod \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\" (UID: \"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c\") "
Oct 30 19:05:34.859449 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.859357    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-credential-socket" pod "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" (UID: "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:34.860044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.859630    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-data" pod "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" (UID: "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:34.860044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.859806    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-workload-socket" pod "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" (UID: "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:34.860044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.860009    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-workload-certs" pod "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" (UID: "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:34.860289 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.860185    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:34.860289 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.860207    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:34.860289 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.860222    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:34.860289 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.860238    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:34.860470 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.860334    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:34.865286 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.865092    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-token" pod "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" (UID: "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:34.865286 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.865220    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-kube-api-access-cnkf6" pod "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" (UID: "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c"). InnerVolumeSpecName "kube-api-access-cnkf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:34.865629 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.865606    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-envoy" pod "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" (UID: "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:34.865814 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.865793    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-podinfo" pod "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" (UID: "add3fc6c-882e-4db7-a2b6-6886cb3bfc5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:05:34.961954 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.961927    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnkf6\" (UniqueName: \"kubernetes.io/projected/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-kube-api-access-cnkf6\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:34.961954 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.961955    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:34.962167 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.961965    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:34.962167 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:34.961974    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:35.192105 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:35.192061    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:05:35.192703 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:35.192679    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:05:35.193596 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:35.193575    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:05:35.195966 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:35.195941    4372 generic.go:356] "Generic (PLEG): container finished" podID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" containerID="8a096cb4590e3de82bd3c9185343f0edc131a2c8a682eaf9268da1d60cc53838" exitCode=0
Oct 30 19:05:35.196064 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:35.195979    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"
Oct 30 19:05:35.196064 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:35.196015    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" event={"ID":"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c","Type":"ContainerDied","Data":"8a096cb4590e3de82bd3c9185343f0edc131a2c8a682eaf9268da1d60cc53838"}
Oct 30 19:05:35.196168 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:35.196063    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8" event={"ID":"add3fc6c-882e-4db7-a2b6-6886cb3bfc5c","Type":"ContainerDied","Data":"34a9378bbb89c81fc7adc3f150196243964cc4ef3cc5ede807cf1f93be8ba4b2"}
Oct 30 19:05:35.196168 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:35.196095    4372 scope.go:122] "RemoveContainer" containerID="8a096cb4590e3de82bd3c9185343f0edc131a2c8a682eaf9268da1d60cc53838"
Oct 30 19:05:35.207508 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:35.207491    4372 scope.go:122] "RemoveContainer" containerID="8a096cb4590e3de82bd3c9185343f0edc131a2c8a682eaf9268da1d60cc53838"
Oct 30 19:05:35.207798 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:05:35.207780    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a096cb4590e3de82bd3c9185343f0edc131a2c8a682eaf9268da1d60cc53838\": container with ID starting with 8a096cb4590e3de82bd3c9185343f0edc131a2c8a682eaf9268da1d60cc53838 not found: ID does not exist" containerID="8a096cb4590e3de82bd3c9185343f0edc131a2c8a682eaf9268da1d60cc53838"
Oct 30 19:05:35.207850 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:35.207806    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a096cb4590e3de82bd3c9185343f0edc131a2c8a682eaf9268da1d60cc53838"} err="failed to get container status \"8a096cb4590e3de82bd3c9185343f0edc131a2c8a682eaf9268da1d60cc53838\": rpc error: code = NotFound desc = could not find container \"8a096cb4590e3de82bd3c9185343f0edc131a2c8a682eaf9268da1d60cc53838\": container with ID starting with 8a096cb4590e3de82bd3c9185343f0edc131a2c8a682eaf9268da1d60cc53838 not found: ID does not exist"
Oct 30 19:05:35.264642 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:35.264608    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"]
Oct 30 19:05:35.271090 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:35.271051    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-vkdv8"]
Oct 30 19:05:36.095231 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.095199    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"]
Oct 30 19:05:36.095909 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.095894    4372 cpu_manager.go:414] "containerMap: removing container" podUID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" containerName="istio-proxy"
Oct 30 19:05:36.095963 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.095911    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" containerName="istio-proxy"
Oct 30 19:05:36.096016 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.096007    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" containerName="istio-proxy"
Oct 30 19:05:36.101137 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.101106    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.111209 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.111188    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4e94aa-openshift-default-dockercfg-ffhn5\""
Oct 30 19:05:36.115484 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.115459    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"]
Oct 30 19:05:36.202151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.185535    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-data\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.202151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.185590    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-credential-socket\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.202151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.185650    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-workload-certs\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.202151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.185691    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istiod-ca-cert\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.202151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.185768    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-workload-socket\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.202151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.185871    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-podinfo\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.202151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.185906    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-token\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.202151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.186253    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-envoy\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.202151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.186297    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s44wk\" (UniqueName: \"kubernetes.io/projected/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-kube-api-access-s44wk\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.407991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.404336    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-podinfo\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.407991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.404427    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-token\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.407991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.404733    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-envoy\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.407991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.404778    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s44wk\" (UniqueName: \"kubernetes.io/projected/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-kube-api-access-s44wk\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.407991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.404890    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-data\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.407991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.404940    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-credential-socket\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.407991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.405059    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-workload-certs\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.407991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.405190    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istiod-ca-cert\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.407991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.405305    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-workload-socket\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.407991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.406578    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-data\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.434854 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.433607    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-credential-socket\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.434854 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.433929    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-workload-certs\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.434854 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.434649    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istiod-ca-cert\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.437840 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.434944    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-workload-socket\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.444063 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.444000    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-podinfo\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.485452 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.460483    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-token\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.519148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.504609    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-envoy\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.519148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.509789    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s44wk\" (UniqueName: \"kubernetes.io/projected/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-kube-api-access-s44wk\") pod \"e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") " pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:36.519148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.512969    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add3fc6c-882e-4db7-a2b6-6886cb3bfc5c" path="/var/lib/kubelet/pods/add3fc6c-882e-4db7-a2b6-6886cb3bfc5c/volumes"
Oct 30 19:05:36.718348 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:36.718253    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:37.039099 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:05:37.039063    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9313e8c4_ec9e_4735_b58a_ce18fa0a692b.slice/crio-1de6da698bc66de6d6b0b63ca4382be5dbcbfc8fd8ef8624cb656ce5f6a64d83 WatchSource:0}: Error finding container 1de6da698bc66de6d6b0b63ca4382be5dbcbfc8fd8ef8624cb656ce5f6a64d83: Status 404 returned error can't find the container with id 1de6da698bc66de6d6b0b63ca4382be5dbcbfc8fd8ef8624cb656ce5f6a64d83
Oct 30 19:05:37.039270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.039240    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"]
Oct 30 19:05:37.042812 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.042772    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:05:37.043041 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.042866    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:05:37.043041 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.042934    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:05:37.277195 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.277167    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 19:05:37.438908 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.438877    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-credential-socket\") pod \"f5de6051-31b1-4529-8db0-d6cb1320a743\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") "
Oct 30 19:05:37.439072 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.438952    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-envoy\") pod \"f5de6051-31b1-4529-8db0-d6cb1320a743\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") "
Oct 30 19:05:37.439072 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.439002    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-podinfo\") pod \"f5de6051-31b1-4529-8db0-d6cb1320a743\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") "
Oct 30 19:05:37.439247 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.439166    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5de6051-31b1-4529-8db0-d6cb1320a743-kube-api-access-4sfn7\" (UniqueName: \"kubernetes.io/projected/f5de6051-31b1-4529-8db0-d6cb1320a743-kube-api-access-4sfn7\") pod \"f5de6051-31b1-4529-8db0-d6cb1320a743\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") "
Oct 30 19:05:37.439247 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.439223    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-credential-socket" pod "f5de6051-31b1-4529-8db0-d6cb1320a743" (UID: "f5de6051-31b1-4529-8db0-d6cb1320a743"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:37.439375 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.439248    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5de6051-31b1-4529-8db0-d6cb1320a743-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5de6051-31b1-4529-8db0-d6cb1320a743-istiod-ca-cert\") pod \"f5de6051-31b1-4529-8db0-d6cb1320a743\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") "
Oct 30 19:05:37.439440 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.439395    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-workload-socket\") pod \"f5de6051-31b1-4529-8db0-d6cb1320a743\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") "
Oct 30 19:05:37.439492 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.439454    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-workload-certs\") pod \"f5de6051-31b1-4529-8db0-d6cb1320a743\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") "
Oct 30 19:05:37.439589 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.439566    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-data\") pod \"f5de6051-31b1-4529-8db0-d6cb1320a743\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") "
Oct 30 19:05:37.439680 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.439658    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-token\" (UniqueName: \"kubernetes.io/projected/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-token\") pod \"f5de6051-31b1-4529-8db0-d6cb1320a743\" (UID: \"f5de6051-31b1-4529-8db0-d6cb1320a743\") "
Oct 30 19:05:37.439680 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.439663    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5de6051-31b1-4529-8db0-d6cb1320a743-istiod-ca-cert" pod "f5de6051-31b1-4529-8db0-d6cb1320a743" (UID: "f5de6051-31b1-4529-8db0-d6cb1320a743"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:37.439885 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.439832    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-workload-socket" pod "f5de6051-31b1-4529-8db0-d6cb1320a743" (UID: "f5de6051-31b1-4529-8db0-d6cb1320a743"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:37.439885 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.439845    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-data" pod "f5de6051-31b1-4529-8db0-d6cb1320a743" (UID: "f5de6051-31b1-4529-8db0-d6cb1320a743"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:37.439885 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.439875    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-workload-certs" pod "f5de6051-31b1-4529-8db0-d6cb1320a743" (UID: "f5de6051-31b1-4529-8db0-d6cb1320a743"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:37.440424 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.440401    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5de6051-31b1-4529-8db0-d6cb1320a743-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:37.440555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.440428    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:37.440555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.440442    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:37.440555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.440457    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:37.440555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.440469    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:37.442725 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.442688    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-podinfo" pod "f5de6051-31b1-4529-8db0-d6cb1320a743" (UID: "f5de6051-31b1-4529-8db0-d6cb1320a743"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:05:37.442725 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.442695    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-envoy" pod "f5de6051-31b1-4529-8db0-d6cb1320a743" (UID: "f5de6051-31b1-4529-8db0-d6cb1320a743"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:37.442893 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.442810    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5de6051-31b1-4529-8db0-d6cb1320a743-kube-api-access-4sfn7" pod "f5de6051-31b1-4529-8db0-d6cb1320a743" (UID: "f5de6051-31b1-4529-8db0-d6cb1320a743"). InnerVolumeSpecName "kube-api-access-4sfn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:37.443839 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.443810    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-token" pod "f5de6051-31b1-4529-8db0-d6cb1320a743" (UID: "f5de6051-31b1-4529-8db0-d6cb1320a743"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:37.463852 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.463815    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" event={"ID":"9313e8c4-ec9e-4735-b58a-ce18fa0a692b","Type":"ContainerStarted","Data":"04a5f256ae5398fcb41558cdad9e438580b3aaacdd46eb94dd7ec1356d055057"}
Oct 30 19:05:37.463994 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.463886    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" event={"ID":"9313e8c4-ec9e-4735-b58a-ce18fa0a692b","Type":"ContainerStarted","Data":"1de6da698bc66de6d6b0b63ca4382be5dbcbfc8fd8ef8624cb656ce5f6a64d83"}
Oct 30 19:05:37.465519 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.465493    4372 generic.go:356] "Generic (PLEG): container finished" podID="f5de6051-31b1-4529-8db0-d6cb1320a743" containerID="dbe1624a94eafa2e8363f87d6ef62d97bb15e1be8cfa1eaa7c113031f6d9e674" exitCode=0
Oct 30 19:05:37.465621 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.465557    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"
Oct 30 19:05:37.465621 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.465571    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54" event={"ID":"f5de6051-31b1-4529-8db0-d6cb1320a743","Type":"ContainerDied","Data":"dbe1624a94eafa2e8363f87d6ef62d97bb15e1be8cfa1eaa7c113031f6d9e674"}
Oct 30 19:05:37.465712 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.465628    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54" event={"ID":"f5de6051-31b1-4529-8db0-d6cb1320a743","Type":"ContainerDied","Data":"9754d265191d6061bb5b76d0a7d5c08530626625d9a805c60f2f73871daec623"}
Oct 30 19:05:37.465712 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.465657    4372 scope.go:122] "RemoveContainer" containerID="dbe1624a94eafa2e8363f87d6ef62d97bb15e1be8cfa1eaa7c113031f6d9e674"
Oct 30 19:05:37.477402 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.476384    4372 scope.go:122] "RemoveContainer" containerID="dbe1624a94eafa2e8363f87d6ef62d97bb15e1be8cfa1eaa7c113031f6d9e674"
Oct 30 19:05:37.477402 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:05:37.476662    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbe1624a94eafa2e8363f87d6ef62d97bb15e1be8cfa1eaa7c113031f6d9e674\": container with ID starting with dbe1624a94eafa2e8363f87d6ef62d97bb15e1be8cfa1eaa7c113031f6d9e674 not found: ID does not exist" containerID="dbe1624a94eafa2e8363f87d6ef62d97bb15e1be8cfa1eaa7c113031f6d9e674"
Oct 30 19:05:37.477402 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.476693    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe1624a94eafa2e8363f87d6ef62d97bb15e1be8cfa1eaa7c113031f6d9e674"} err="failed to get container status \"dbe1624a94eafa2e8363f87d6ef62d97bb15e1be8cfa1eaa7c113031f6d9e674\": rpc error: code = NotFound desc = could not find container \"dbe1624a94eafa2e8363f87d6ef62d97bb15e1be8cfa1eaa7c113031f6d9e674\": container with ID starting with dbe1624a94eafa2e8363f87d6ef62d97bb15e1be8cfa1eaa7c113031f6d9e674 not found: ID does not exist"
Oct 30 19:05:37.490466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.490419    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" podStartSLOduration=1.4904024470000001 podStartE2EDuration="1.490402447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:37.488524235 +0000 UTC m=+1951.356942573" watchObservedRunningTime="2026-10-30 19:05:37.490402447 +0000 UTC m=+1951.358820777"
Oct 30 19:05:37.507117 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.507091    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"]
Oct 30 19:05:37.515859 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.515837    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-6vk54"]
Oct 30 19:05:37.541956 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.541933    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sfn7\" (UniqueName: \"kubernetes.io/projected/f5de6051-31b1-4529-8db0-d6cb1320a743-kube-api-access-4sfn7\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:37.542147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.542106    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:37.542252 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.542152    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:37.542252 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.542172    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5de6051-31b1-4529-8db0-d6cb1320a743-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:37.718981 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:37.718910    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:38.267596 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:38.267559    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5de6051-31b1-4529-8db0-d6cb1320a743" path="/var/lib/kubelet/pods/f5de6051-31b1-4529-8db0-d6cb1320a743/volumes"
Oct 30 19:05:38.724868 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:38.724839    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:39.513151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:39.511633    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:39.522408 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:39.522376    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:05:41.083510 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.083477    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"]
Oct 30 19:05:41.084215 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.084198    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f5de6051-31b1-4529-8db0-d6cb1320a743" containerName="istio-proxy"
Oct 30 19:05:41.084284 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.084216    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5de6051-31b1-4529-8db0-d6cb1320a743" containerName="istio-proxy"
Oct 30 19:05:41.084361 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.084350    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5de6051-31b1-4529-8db0-d6cb1320a743" containerName="istio-proxy"
Oct 30 19:05:41.089522 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.089505    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"
Oct 30 19:05:41.098555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.098531    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-4e94aa\""
Oct 30 19:05:41.102267 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.102244    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"]
Oct 30 19:05:41.107943 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.107918    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-4e94aa-dockercfg-tmtcg\""
Oct 30 19:05:41.134463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.134431    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"]
Oct 30 19:05:41.139989 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.139968    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"
Oct 30 19:05:41.149429 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.149403    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"]
Oct 30 19:05:41.334397 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.240485    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96p5j\" (UniqueName: \"kubernetes.io/projected/f681d0ac-6413-4ccb-88ba-3e0e278bd663-kube-api-access-96p5j\") pod \"payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4\" (UID: \"f681d0ac-6413-4ccb-88ba-3e0e278bd663\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"
Oct 30 19:05:41.334397 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.240643    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f681d0ac-6413-4ccb-88ba-3e0e278bd663-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4\" (UID: \"f681d0ac-6413-4ccb-88ba-3e0e278bd663\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"
Oct 30 19:05:41.334397 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.240820    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f57597b-6f1c-47e2-9891-5432b04e8fb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6\" (UID: \"4f57597b-6f1c-47e2-9891-5432b04e8fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"
Oct 30 19:05:41.334397 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.240868    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mdfc\" (UniqueName: \"kubernetes.io/projected/4f57597b-6f1c-47e2-9891-5432b04e8fb1-kube-api-access-4mdfc\") pod \"payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6\" (UID: \"4f57597b-6f1c-47e2-9891-5432b04e8fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"
Oct 30 19:05:41.378371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.362194    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96p5j\" (UniqueName: \"kubernetes.io/projected/f681d0ac-6413-4ccb-88ba-3e0e278bd663-kube-api-access-96p5j\") pod \"payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4\" (UID: \"f681d0ac-6413-4ccb-88ba-3e0e278bd663\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"
Oct 30 19:05:41.378371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.362310    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f681d0ac-6413-4ccb-88ba-3e0e278bd663-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4\" (UID: \"f681d0ac-6413-4ccb-88ba-3e0e278bd663\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"
Oct 30 19:05:41.378371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.362477    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f57597b-6f1c-47e2-9891-5432b04e8fb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6\" (UID: \"4f57597b-6f1c-47e2-9891-5432b04e8fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"
Oct 30 19:05:41.378371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.362515    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mdfc\" (UniqueName: \"kubernetes.io/projected/4f57597b-6f1c-47e2-9891-5432b04e8fb1-kube-api-access-4mdfc\") pod \"payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6\" (UID: \"4f57597b-6f1c-47e2-9891-5432b04e8fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"
Oct 30 19:05:41.378371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.366505    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f57597b-6f1c-47e2-9891-5432b04e8fb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6\" (UID: \"4f57597b-6f1c-47e2-9891-5432b04e8fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"
Oct 30 19:05:41.379402 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.379370    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f681d0ac-6413-4ccb-88ba-3e0e278bd663-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4\" (UID: \"f681d0ac-6413-4ccb-88ba-3e0e278bd663\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"
Oct 30 19:05:41.398696 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.398664    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mdfc\" (UniqueName: \"kubernetes.io/projected/4f57597b-6f1c-47e2-9891-5432b04e8fb1-kube-api-access-4mdfc\") pod \"payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6\" (UID: \"4f57597b-6f1c-47e2-9891-5432b04e8fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"
Oct 30 19:05:41.404710 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.404683    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"
Oct 30 19:05:41.414779 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.414753    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96p5j\" (UniqueName: \"kubernetes.io/projected/f681d0ac-6413-4ccb-88ba-3e0e278bd663-kube-api-access-96p5j\") pod \"payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4\" (UID: \"f681d0ac-6413-4ccb-88ba-3e0e278bd663\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"
Oct 30 19:05:41.477209 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.466712    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"
Oct 30 19:05:41.813228 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:05:41.813197    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f57597b_6f1c_47e2_9891_5432b04e8fb1.slice/crio-6276cf05c3bc622a26a8318156ac7142a0a325169597bd48d74fcf30854ed1dd WatchSource:0}: Error finding container 6276cf05c3bc622a26a8318156ac7142a0a325169597bd48d74fcf30854ed1dd: Status 404 returned error can't find the container with id 6276cf05c3bc622a26a8318156ac7142a0a325169597bd48d74fcf30854ed1dd
Oct 30 19:05:41.814593 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.814565    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"]
Oct 30 19:05:41.827163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:41.827120    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"]
Oct 30 19:05:41.828078 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:05:41.828053    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf681d0ac_6413_4ccb_88ba_3e0e278bd663.slice/crio-f07c0660e10deb02b7a42e7439d3e2be8107639e43692cd6593a3a118054ec97 WatchSource:0}: Error finding container f07c0660e10deb02b7a42e7439d3e2be8107639e43692cd6593a3a118054ec97: Status 404 returned error can't find the container with id f07c0660e10deb02b7a42e7439d3e2be8107639e43692cd6593a3a118054ec97
Oct 30 19:05:42.552229 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.552184    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4" event={"ID":"f681d0ac-6413-4ccb-88ba-3e0e278bd663","Type":"ContainerStarted","Data":"2c4cd7e63cb5481b2a01cc05ae4d1aa55c721d91ac91b2487e9152921e5dd469"}
Oct 30 19:05:42.552766 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.552249    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4" event={"ID":"f681d0ac-6413-4ccb-88ba-3e0e278bd663","Type":"ContainerStarted","Data":"f07c0660e10deb02b7a42e7439d3e2be8107639e43692cd6593a3a118054ec97"}
Oct 30 19:05:42.552766 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.552278    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"
Oct 30 19:05:42.553816 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.553790    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6" event={"ID":"4f57597b-6f1c-47e2-9891-5432b04e8fb1","Type":"ContainerStarted","Data":"ef9d8561830b85bb2aa80545eb768243264c44200dca730cf3e72ec52c27edda"}
Oct 30 19:05:42.553944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.553841    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6" event={"ID":"4f57597b-6f1c-47e2-9891-5432b04e8fb1","Type":"ContainerStarted","Data":"6276cf05c3bc622a26a8318156ac7142a0a325169597bd48d74fcf30854ed1dd"}
Oct 30 19:05:42.553944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.553882    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"
Oct 30 19:05:42.573114 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.572953    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4" podStartSLOduration=1.5729396759999998 podStartE2EDuration="1.572939676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:42.572198455 +0000 UTC m=+1956.440616786" watchObservedRunningTime="2026-10-30 19:05:42.572939676 +0000 UTC m=+1956.441358007"
Oct 30 19:05:42.594457 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.594411    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6" podStartSLOduration=1.594395955 podStartE2EDuration="1.594395955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:42.592091978 +0000 UTC m=+1956.460510308" watchObservedRunningTime="2026-10-30 19:05:42.594395955 +0000 UTC m=+1956.462814286"
Oct 30 19:05:42.957651 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.957620    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"]
Oct 30 19:05:42.962301 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.962275    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:42.969578 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.969550    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c9e9de\"/\"default-dockercfg-nrhwv\""
Oct 30 19:05:42.969793 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.969659    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c9e9de\"/\"autores-model-c9e9de-kserve-self-signed-certs\""
Oct 30 19:05:42.974209 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.974188    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c9e9de\"/\"kube-root-ca.crt\""
Oct 30 19:05:42.974329 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.974208    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c9e9de\"/\"openshift-service-ca.crt\""
Oct 30 19:05:42.976067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.976047    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"]
Oct 30 19:05:42.993343 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.993317    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcxvx\" (UniqueName: \"kubernetes.io/projected/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-kube-api-access-lcxvx\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:42.993451 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.993382    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-model-cache\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:42.993548 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.993453    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-tls-certs\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:42.993548 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.993542    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-tmp-dir\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:42.993661 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.993607    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-home\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:42.993661 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:42.993652    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-dshm\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:43.094954 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:43.094925    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-tmp-dir\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:43.095070 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:43.095004    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-home\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:43.095114 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:43.095078    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-dshm\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:43.095205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:43.095184    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcxvx\" (UniqueName: \"kubernetes.io/projected/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-kube-api-access-lcxvx\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:43.095270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:43.095252    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-model-cache\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:43.095364 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:43.095349    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-tls-certs\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:43.095416 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:43.095394    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-tmp-dir\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:43.095477 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:43.095428    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-home\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:43.095558 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:43.095542    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-model-cache\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:43.097548 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:43.097528    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-dshm\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:43.098053 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:43.098030    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-tls-certs\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:43.108042 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:43.108020    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcxvx\" (UniqueName: \"kubernetes.io/projected/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-kube-api-access-lcxvx\") pod \"autores-model-c9e9de-kserve-dbb5fc985-xt7x8\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") " pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:43.279288 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:43.279188    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:43.587481 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:43.587444    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"]
Oct 30 19:05:44.573390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:44.573348    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8" event={"ID":"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c","Type":"ContainerStarted","Data":"4478bca04df5854fa8fd4305a190f2c107d8bbc2a370a539dc2060eceb562ddc"}
Oct 30 19:05:44.573566 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:44.573412    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8" event={"ID":"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c","Type":"ContainerStarted","Data":"cc78031701cf9cac2f8fccf47d333413b160af25649061f92d225df0bfc8e99e"}
Oct 30 19:05:44.602067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:44.602019    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8" podStartSLOduration=2.602005005 podStartE2EDuration="2.602005005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:44.599954619 +0000 UTC m=+1958.468372951" watchObservedRunningTime="2026-10-30 19:05:44.602005005 +0000 UTC m=+1958.470423344"
Oct 30 19:05:45.279968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:45.279935    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:53.296153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:53.292391    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:53.321363 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:53.321336    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:53.568768 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:53.568679    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"
Oct 30 19:05:53.574156 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:53.573901    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"
Oct 30 19:05:53.735344 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:53.735310    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:05:57.725521 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:05:57.725491    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:05:59.117379 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.117346    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"]
Oct 30 19:05:59.121898 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.121878    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"
Oct 30 19:05:59.129484 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.129459    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-7c6027\""
Oct 30 19:05:59.136266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.136242    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"]
Oct 30 19:05:59.156071 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.156041    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"]
Oct 30 19:05:59.160363 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.160342    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"
Oct 30 19:05:59.166385 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.166365    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-7c6027-dockercfg-h7bvj\""
Oct 30 19:05:59.172446 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.172419    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"]
Oct 30 19:05:59.251351 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.251317    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cj8c\" (UniqueName: \"kubernetes.io/projected/387edd5f-c28a-4732-ad0a-6af167143726-kube-api-access-2cj8c\") pod \"payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm\" (UID: \"387edd5f-c28a-4732-ad0a-6af167143726\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"
Oct 30 19:05:59.251529 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.251356    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brmrk\" (UniqueName: \"kubernetes.io/projected/a48daad3-b6ea-4e65-87dc-03a1c811a2cb-kube-api-access-brmrk\") pod \"payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn\" (UID: \"a48daad3-b6ea-4e65-87dc-03a1c811a2cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"
Oct 30 19:05:59.251629 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.251607    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387edd5f-c28a-4732-ad0a-6af167143726-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm\" (UID: \"387edd5f-c28a-4732-ad0a-6af167143726\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"
Oct 30 19:05:59.251698 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.251665    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48daad3-b6ea-4e65-87dc-03a1c811a2cb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn\" (UID: \"a48daad3-b6ea-4e65-87dc-03a1c811a2cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"
Oct 30 19:05:59.353030 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.352992    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cj8c\" (UniqueName: \"kubernetes.io/projected/387edd5f-c28a-4732-ad0a-6af167143726-kube-api-access-2cj8c\") pod \"payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm\" (UID: \"387edd5f-c28a-4732-ad0a-6af167143726\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"
Oct 30 19:05:59.353265 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.353045    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmrk\" (UniqueName: \"kubernetes.io/projected/a48daad3-b6ea-4e65-87dc-03a1c811a2cb-kube-api-access-brmrk\") pod \"payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn\" (UID: \"a48daad3-b6ea-4e65-87dc-03a1c811a2cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"
Oct 30 19:05:59.353358 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.353338    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387edd5f-c28a-4732-ad0a-6af167143726-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm\" (UID: \"387edd5f-c28a-4732-ad0a-6af167143726\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"
Oct 30 19:05:59.353453 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.353397    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48daad3-b6ea-4e65-87dc-03a1c811a2cb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn\" (UID: \"a48daad3-b6ea-4e65-87dc-03a1c811a2cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"
Oct 30 19:05:59.354097 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.354071    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387edd5f-c28a-4732-ad0a-6af167143726-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm\" (UID: \"387edd5f-c28a-4732-ad0a-6af167143726\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"
Oct 30 19:05:59.354314 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.354292    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48daad3-b6ea-4e65-87dc-03a1c811a2cb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn\" (UID: \"a48daad3-b6ea-4e65-87dc-03a1c811a2cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"
Oct 30 19:05:59.372803 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.372744    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmrk\" (UniqueName: \"kubernetes.io/projected/a48daad3-b6ea-4e65-87dc-03a1c811a2cb-kube-api-access-brmrk\") pod \"payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn\" (UID: \"a48daad3-b6ea-4e65-87dc-03a1c811a2cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"
Oct 30 19:05:59.376875 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.376852    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cj8c\" (UniqueName: \"kubernetes.io/projected/387edd5f-c28a-4732-ad0a-6af167143726-kube-api-access-2cj8c\") pod \"payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm\" (UID: \"387edd5f-c28a-4732-ad0a-6af167143726\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"
Oct 30 19:05:59.444251 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.444221    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"
Oct 30 19:05:59.481844 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:05:59.481814    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"
Oct 30 19:06:00.293600 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:06:00.293568    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387edd5f_c28a_4732_ad0a_6af167143726.slice/crio-39887f82698247bb7d399e0590c731e2b22ff20a686a094d11b0c45c84f14b5b WatchSource:0}: Error finding container 39887f82698247bb7d399e0590c731e2b22ff20a686a094d11b0c45c84f14b5b: Status 404 returned error can't find the container with id 39887f82698247bb7d399e0590c731e2b22ff20a686a094d11b0c45c84f14b5b
Oct 30 19:06:00.294000 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.293797    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"]
Oct 30 19:06:00.379854 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:06:00.379824    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48daad3_b6ea_4e65_87dc_03a1c811a2cb.slice/crio-15ab5b257f673ae5ea69384a28a4698f25f98ad29c64e9e45010e6c01657b01c WatchSource:0}: Error finding container 15ab5b257f673ae5ea69384a28a4698f25f98ad29c64e9e45010e6c01657b01c: Status 404 returned error can't find the container with id 15ab5b257f673ae5ea69384a28a4698f25f98ad29c64e9e45010e6c01657b01c
Oct 30 19:06:00.380956 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.380929    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"]
Oct 30 19:06:00.503535 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.503497    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"]
Oct 30 19:06:00.508847 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.508812    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:00.525855 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.522341    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"]
Oct 30 19:06:00.530490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.530455    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"agc-cred-54fc64a67d3d\""
Oct 30 19:06:00.530773 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.530544    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d72beb\""
Oct 30 19:06:00.530773 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.530572    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"kube-root-ca.crt\""
Oct 30 19:06:00.530773 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.530619    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"routing-overlay\""
Oct 30 19:06:00.530773 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.530634    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"openshift-service-ca.crt\""
Oct 30 19:06:00.531074 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.530776    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"payload-processing-external-model-e2e--b99c0f48-dockercfg-hq7ks\""
Oct 30 19:06:00.574426 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.574390    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:00.574612 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.574464    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:00.574612 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.574580    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29zs4\" (UniqueName: \"kubernetes.io/projected/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-kube-api-access-29zs4\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:00.574739 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.574708    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:00.685153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.678784    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:00.685153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.678887    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:00.685153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.678956    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29zs4\" (UniqueName: \"kubernetes.io/projected/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-kube-api-access-29zs4\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:00.685153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.679051    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:00.685153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.679985    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:00.685153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.680729    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:00.753526 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.735639    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29zs4\" (UniqueName: \"kubernetes.io/projected/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-kube-api-access-29zs4\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:00.776374 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.775471    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:00.842173 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:00.841507    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:01.078728 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:01.078594    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm" event={"ID":"387edd5f-c28a-4732-ad0a-6af167143726","Type":"ContainerStarted","Data":"2ef2bc82034a2e592b05bd1060d996dfdbba9e823d6c0d58f38bd5310e2ea959"}
Oct 30 19:06:01.078728 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:01.078659    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm" event={"ID":"387edd5f-c28a-4732-ad0a-6af167143726","Type":"ContainerStarted","Data":"39887f82698247bb7d399e0590c731e2b22ff20a686a094d11b0c45c84f14b5b"}
Oct 30 19:06:01.078728 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:01.078698    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"
Oct 30 19:06:01.080905 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:01.080879    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn" event={"ID":"a48daad3-b6ea-4e65-87dc-03a1c811a2cb","Type":"ContainerStarted","Data":"4e800aee892fbcca4a611a51e12f827471d6dba87e178de1629bfda73682c6b8"}
Oct 30 19:06:01.081025 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:01.080930    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn" event={"ID":"a48daad3-b6ea-4e65-87dc-03a1c811a2cb","Type":"ContainerStarted","Data":"15ab5b257f673ae5ea69384a28a4698f25f98ad29c64e9e45010e6c01657b01c"}
Oct 30 19:06:01.081940 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:01.081916    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"
Oct 30 19:06:01.101894 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:01.101830    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm" podStartSLOduration=2.101816267 podStartE2EDuration="2.101816267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:01.099284859 +0000 UTC m=+1974.967703190" watchObservedRunningTime="2026-10-30 19:06:01.101816267 +0000 UTC m=+1974.970234597"
Oct 30 19:06:01.129503 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:01.129433    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn" podStartSLOduration=2.12941584 podStartE2EDuration="2.12941584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:01.124745123 +0000 UTC m=+1974.993163453" watchObservedRunningTime="2026-10-30 19:06:01.12941584 +0000 UTC m=+1974.997834171"
Oct 30 19:06:01.317180 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:01.317150    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"]
Oct 30 19:06:01.318714 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:06:01.318687    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ae5b4b_4033_4fe5_93ab_cbdd22757d08.slice/crio-42498adee09fe496e2e9a2f4570e9916b6d6905d322cab0a496a9e8ab4fc7a3f WatchSource:0}: Error finding container 42498adee09fe496e2e9a2f4570e9916b6d6905d322cab0a496a9e8ab4fc7a3f: Status 404 returned error can't find the container with id 42498adee09fe496e2e9a2f4570e9916b6d6905d322cab0a496a9e8ab4fc7a3f
Oct 30 19:06:02.090028 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:02.089989    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q" event={"ID":"82ae5b4b-4033-4fe5-93ab-cbdd22757d08","Type":"ContainerStarted","Data":"7246634a8465a1923f6d75084aa6481c2557acb27b906aaecdfe3ed28eda73ac"}
Oct 30 19:06:02.090264 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:02.090056    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q" event={"ID":"82ae5b4b-4033-4fe5-93ab-cbdd22757d08","Type":"ContainerStarted","Data":"42498adee09fe496e2e9a2f4570e9916b6d6905d322cab0a496a9e8ab4fc7a3f"}
Oct 30 19:06:02.127194 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:02.127142    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q" podStartSLOduration=2.127111892 podStartE2EDuration="2.127111892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:02.123120484 +0000 UTC m=+1975.991538816" watchObservedRunningTime="2026-10-30 19:06:02.127111892 +0000 UTC m=+1975.995530221"
Oct 30 19:06:03.099466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:03.099087    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:03.185276 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:03.185235    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:06:03.187373 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:03.185293    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:06:04.001970 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:04.001935    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"]
Oct 30 19:06:04.169793 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:04.168987    4372 generic.go:356] "Generic (PLEG): container finished" podID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerID="c1d6b783d5f6c51bb463972000e9bc0b29fa67bd71004c563bde6a647dcbe247" exitCode=0
Oct 30 19:06:04.170367 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:04.170300    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" event={"ID":"a6b96f6e-b871-499b-ab8f-18fffaa3ece2","Type":"ContainerDied","Data":"c1d6b783d5f6c51bb463972000e9bc0b29fa67bd71004c563bde6a647dcbe247"}
Oct 30 19:06:04.178174 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:04.170773    4372 scope.go:122] "RemoveContainer" containerID="c1d6b783d5f6c51bb463972000e9bc0b29fa67bd71004c563bde6a647dcbe247"
Oct 30 19:06:04.190149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:04.187938    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:04.190149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:04.188046    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:04.190149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:04.188098    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:05.198380 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:05.196683    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" event={"ID":"a6b96f6e-b871-499b-ab8f-18fffaa3ece2","Type":"ContainerStarted","Data":"34914257f0fe610ae9da5ecb927e9b5e4e57f23f1acde2a2d9b94e875c1345a1"}
Oct 30 19:06:05.297437 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:05.297404    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:06:05.297615 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:05.297546    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:06:05.297615 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:05.297601    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:06:06.299621 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:06.299578    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:06:06.300284 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:06.300247    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:06:07.299506 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:07.299467    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:06:07.299807 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:07.299776    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:06:08.303747 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:08.303708    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:06:08.304409 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:08.304371    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:06:09.303591 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:09.303506    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:06:09.303591 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:09.303573    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:06:10.301498 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:10.301463    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:06:10.302106 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:10.302076    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:06:11.301562 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:11.301527    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:06:11.302042 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:11.301801    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:06:11.302497 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:11.302479    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:06:12.093835 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.093807    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"
Oct 30 19:06:12.096044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.096016    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"]
Oct 30 19:06:12.102327 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.102307    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.109298 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.109260    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f7c6e3-openshift-default-dockercfg-9qn52\""
Oct 30 19:06:12.114443 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.114422    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"]
Oct 30 19:06:12.240906 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.240864    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-credential-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.241169 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.241061    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-istio-envoy\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.241355 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.241300    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6668392b-be79-447f-b009-907ff82e0420-istio-token\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.241440 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.241426    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6668392b-be79-447f-b009-907ff82e0420-istiod-ca-cert\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.241486 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.241467    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-workload-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.241538 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.241498    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-istio-data\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.241606 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.241576    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6668392b-be79-447f-b009-907ff82e0420-istio-podinfo\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.241737 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.241707    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-workload-certs\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.241835 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.241767    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvsb\" (UniqueName: \"kubernetes.io/projected/6668392b-be79-447f-b009-907ff82e0420-kube-api-access-dnvsb\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.343347 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.343313    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6668392b-be79-447f-b009-907ff82e0420-istio-token\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.343797 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.343444    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6668392b-be79-447f-b009-907ff82e0420-istiod-ca-cert\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.343797 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.343510    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-workload-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.343797 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.343548    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-istio-data\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.343797 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.343584    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6668392b-be79-447f-b009-907ff82e0420-istio-podinfo\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.343797 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.343741    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-workload-certs\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.343797 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.343795    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvsb\" (UniqueName: \"kubernetes.io/projected/6668392b-be79-447f-b009-907ff82e0420-kube-api-access-dnvsb\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.344197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.343862    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-credential-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.344197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.343942    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-istio-envoy\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.344339 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.344248    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-istio-data\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.344339 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.344286    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-workload-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.344462 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.344342    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-workload-certs\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.344553 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.344532    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-credential-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.344940 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.344916    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6668392b-be79-447f-b009-907ff82e0420-istiod-ca-cert\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.346641 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.346614    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6668392b-be79-447f-b009-907ff82e0420-istio-podinfo\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.346839 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.346821    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-istio-envoy\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.356263 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.356238    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6668392b-be79-447f-b009-907ff82e0420-istio-token\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.357318 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.357298    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvsb\" (UniqueName: \"kubernetes.io/projected/6668392b-be79-447f-b009-907ff82e0420-kube-api-access-dnvsb\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.418158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.418116    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:12.714143 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:06:12.714102    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6668392b_be79_447f_b009_907ff82e0420.slice/crio-b378cb7f03e88f95ef852e987687c877032d433fef9cc0737659f47bb10d8dcc WatchSource:0}: Error finding container b378cb7f03e88f95ef852e987687c877032d433fef9cc0737659f47bb10d8dcc: Status 404 returned error can't find the container with id b378cb7f03e88f95ef852e987687c877032d433fef9cc0737659f47bb10d8dcc
Oct 30 19:06:12.714366 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.714344    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"]
Oct 30 19:06:12.717110 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.717077    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:12.717248 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.717190    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:12.717317 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:12.717244    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:13.103577 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:13.103548    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"
Oct 30 19:06:13.265076 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:13.265040    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597" event={"ID":"6668392b-be79-447f-b009-907ff82e0420","Type":"ContainerStarted","Data":"60dbcca6af1b5fe7ffeb44760ddfc842f4b2b0ef86fe2cfdcfd7297efc241d7b"}
Oct 30 19:06:13.265297 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:13.265097    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597" event={"ID":"6668392b-be79-447f-b009-907ff82e0420","Type":"ContainerStarted","Data":"b378cb7f03e88f95ef852e987687c877032d433fef9cc0737659f47bb10d8dcc"}
Oct 30 19:06:13.295399 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:13.295344    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597" podStartSLOduration=1.295328985 podStartE2EDuration="1.295328985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:13.290004491 +0000 UTC m=+1987.158422824" watchObservedRunningTime="2026-10-30 19:06:13.295328985 +0000 UTC m=+1987.163747315"
Oct 30 19:06:13.418779 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:13.418669    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:13.421248 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:13.421203    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:06:13.421434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:13.421269    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597" podUID="6668392b-be79-447f-b009-907ff82e0420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 19:06:14.173220 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:14.173189    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:06:14.418787 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:14.418757    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:06:14.419248 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:14.419218    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597" podUID="6668392b-be79-447f-b009-907ff82e0420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 19:06:15.207167 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.207080    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"]
Oct 30 19:06:15.213100 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.213070    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.220222 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.220195    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"]
Oct 30 19:06:15.221419 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.221393    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-de4bda-openshift-default-dockercfg-jchfn\""
Oct 30 19:06:15.322754 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.322716    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.322938 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.322775    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.322938 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.322870    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.322938 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.322916    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.323051 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.322982    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.323051 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.323040    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.323141 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.323108    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j85jt\" (UniqueName: \"kubernetes.io/projected/cf21565a-4381-4ed0-ad6c-015853c23d1e-kube-api-access-j85jt\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.323197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.323181    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf21565a-4381-4ed0-ad6c-015853c23d1e-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.323236 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.323202    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.428161 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.425995    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:06:15.428161 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.426090    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597" podUID="6668392b-be79-447f-b009-907ff82e0420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 19:06:15.428161 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.426284    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.428161 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.426362    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.428161 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.426403    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.428161 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.426473    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.428161 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.426510    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.428161 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.426648    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.428161 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.426860    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j85jt\" (UniqueName: \"kubernetes.io/projected/cf21565a-4381-4ed0-ad6c-015853c23d1e-kube-api-access-j85jt\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.428161 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.426946    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf21565a-4381-4ed0-ad6c-015853c23d1e-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.428161 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.426986    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.429359 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.427993    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.430179 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.430145    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.435153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.431079    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf21565a-4381-4ed0-ad6c-015853c23d1e-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.435153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.431613    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.435806 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.435782    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.470111 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.469997    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.470912 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.470856    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.494154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.490890    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j85jt\" (UniqueName: \"kubernetes.io/projected/cf21565a-4381-4ed0-ad6c-015853c23d1e-kube-api-access-j85jt\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.494394 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.491767    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.535428 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.535402    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:15.852103 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:06:15.852074    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf21565a_4381_4ed0_ad6c_015853c23d1e.slice/crio-d0d8deba71649bc2612d32b2a79afd47d3c8494cae28b9a5b8fb726045ec25b8 WatchSource:0}: Error finding container d0d8deba71649bc2612d32b2a79afd47d3c8494cae28b9a5b8fb726045ec25b8: Status 404 returned error can't find the container with id d0d8deba71649bc2612d32b2a79afd47d3c8494cae28b9a5b8fb726045ec25b8
Oct 30 19:06:15.852670 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.852636    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"]
Oct 30 19:06:15.854888 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.854857    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:15.854964 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.854919    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:15.854964 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:15.854951    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:16.346059 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:16.346025    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr" event={"ID":"cf21565a-4381-4ed0-ad6c-015853c23d1e","Type":"ContainerStarted","Data":"b8233b629e22056cc8404c2bd383cdef89f87f035f630638b45638ab1495395d"}
Oct 30 19:06:16.346280 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:16.346090    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr" event={"ID":"cf21565a-4381-4ed0-ad6c-015853c23d1e","Type":"ContainerStarted","Data":"d0d8deba71649bc2612d32b2a79afd47d3c8494cae28b9a5b8fb726045ec25b8"}
Oct 30 19:06:16.377632 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:16.377573    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr" podStartSLOduration=1.37755924 podStartE2EDuration="1.37755924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:16.373117576 +0000 UTC m=+1990.241535907" watchObservedRunningTime="2026-10-30 19:06:16.37755924 +0000 UTC m=+1990.245977569"
Oct 30 19:06:16.418956 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:16.418925    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:06:16.419114 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:16.418975    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597" podUID="6668392b-be79-447f-b009-907ff82e0420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 19:06:16.535940 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:16.535906    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:16.537639 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:16.537610    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:06:16.537789 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:16.537662    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr" podUID="cf21565a-4381-4ed0-ad6c-015853c23d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 19:06:17.425148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:17.424531    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:06:17.425148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:17.424596    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597" podUID="6668392b-be79-447f-b009-907ff82e0420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 19:06:17.536599 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:17.536563    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:06:17.537193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:17.537163    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr" podUID="cf21565a-4381-4ed0-ad6c-015853c23d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 19:06:18.421692 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:18.421654    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:06:18.422151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:18.421968    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597" podUID="6668392b-be79-447f-b009-907ff82e0420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 19:06:18.540152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:18.537054    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:06:18.540152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:18.537153    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr" podUID="cf21565a-4381-4ed0-ad6c-015853c23d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 19:06:19.029952 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.029494    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8" podUID="b30f3c4f-5c7a-4766-bc11-7cdb826aef4c" containerName="main" containerID="cri-o://4478bca04df5854fa8fd4305a190f2c107d8bbc2a370a539dc2060eceb562ddc" gracePeriod=45
Oct 30 19:06:19.537108 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.537075    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:06:19.537451 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.537422    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr" podUID="cf21565a-4381-4ed0-ad6c-015853c23d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 19:06:19.541431 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.541318    4372 generic.go:356] "Generic (PLEG): container finished" podID="b30f3c4f-5c7a-4766-bc11-7cdb826aef4c" containerID="4478bca04df5854fa8fd4305a190f2c107d8bbc2a370a539dc2060eceb562ddc" exitCode=0
Oct 30 19:06:19.541431 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.541384    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8" event={"ID":"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c","Type":"ContainerDied","Data":"4478bca04df5854fa8fd4305a190f2c107d8bbc2a370a539dc2060eceb562ddc"}
Oct 30 19:06:19.572843 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.572797    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:06:19.671827 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.671795    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:19.672672 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.672642    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:19.673602 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.673582    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:06:19.709644 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.708897    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-tls-certs\") pod \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") "
Oct 30 19:06:19.709644 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.709015    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-dshm\") pod \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") "
Oct 30 19:06:19.709644 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.709168    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-home\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-home\") pod \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") "
Oct 30 19:06:19.709644 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.709226    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-model-cache\") pod \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") "
Oct 30 19:06:19.709644 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.709305    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-kube-api-access-lcxvx\" (UniqueName: \"kubernetes.io/projected/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-kube-api-access-lcxvx\") pod \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") "
Oct 30 19:06:19.709644 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.709401    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-tmp-dir\") pod \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\" (UID: \"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c\") "
Oct 30 19:06:19.709644 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.709447    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-home" pod "b30f3c4f-5c7a-4766-bc11-7cdb826aef4c" (UID: "b30f3c4f-5c7a-4766-bc11-7cdb826aef4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:19.710186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.709910    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-model-cache" pod "b30f3c4f-5c7a-4766-bc11-7cdb826aef4c" (UID: "b30f3c4f-5c7a-4766-bc11-7cdb826aef4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:19.710186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.710030    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:19.710186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.710043    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:19.710186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.710068    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-tmp-dir" pod "b30f3c4f-5c7a-4766-bc11-7cdb826aef4c" (UID: "b30f3c4f-5c7a-4766-bc11-7cdb826aef4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:19.712854 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.712826    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-tls-certs" pod "b30f3c4f-5c7a-4766-bc11-7cdb826aef4c" (UID: "b30f3c4f-5c7a-4766-bc11-7cdb826aef4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:19.713035 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.712872    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-kube-api-access-lcxvx" pod "b30f3c4f-5c7a-4766-bc11-7cdb826aef4c" (UID: "b30f3c4f-5c7a-4766-bc11-7cdb826aef4c"). InnerVolumeSpecName "kube-api-access-lcxvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:19.713528 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.713485    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-dshm" pod "b30f3c4f-5c7a-4766-bc11-7cdb826aef4c" (UID: "b30f3c4f-5c7a-4766-bc11-7cdb826aef4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:19.811919 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.811884    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcxvx\" (UniqueName: \"kubernetes.io/projected/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-kube-api-access-lcxvx\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:19.811919 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.811916    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:19.812248 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.811931    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:19.812248 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:19.811945    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:20.269029 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.269002    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"]
Oct 30 19:06:20.269926 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.269903    4372 cpu_manager.go:414] "containerMap: removing container" podUID="b30f3c4f-5c7a-4766-bc11-7cdb826aef4c" containerName="main"
Oct 30 19:06:20.270065 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.269927    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30f3c4f-5c7a-4766-bc11-7cdb826aef4c" containerName="main"
Oct 30 19:06:20.270142 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.270102    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30f3c4f-5c7a-4766-bc11-7cdb826aef4c" containerName="main"
Oct 30 19:06:20.273913 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.273893    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.299152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.288631    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-c9e9de-openshift-default-dockercfg-ct2zv\""
Oct 30 19:06:20.334207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.331688    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-workload-certs\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.334207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.331737    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-podinfo\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.334207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.331771    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-workload-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.334207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.331827    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-envoy\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.334207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.331966    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-data\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.334207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.332003    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b1d5ca-641e-46f4-92ab-9e3c95014445-istiod-ca-cert\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.334207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.332038    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-credential-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.334207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.332256    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-token\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.334207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.332434    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmlqt\" (UniqueName: \"kubernetes.io/projected/04b1d5ca-641e-46f4-92ab-9e3c95014445-kube-api-access-bmlqt\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.346339 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.346309    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"]
Oct 30 19:06:20.433899 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.433866    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-workload-certs\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.434103 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.433903    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-podinfo\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.434103 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.433925    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-workload-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.434103 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.433955    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-envoy\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.434103 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.434012    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-data\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.434103 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.434031    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b1d5ca-641e-46f4-92ab-9e3c95014445-istiod-ca-cert\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.434103 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.434050    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-credential-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.434476 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.434157    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-token\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.434476 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.434254    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmlqt\" (UniqueName: \"kubernetes.io/projected/04b1d5ca-641e-46f4-92ab-9e3c95014445-kube-api-access-bmlqt\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.434476 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.434430    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-workload-certs\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.434476 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.434464    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-workload-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.434693 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.434635    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-data\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.434693 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.434660    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-credential-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.435406 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.435377    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b1d5ca-641e-46f4-92ab-9e3c95014445-istiod-ca-cert\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.437203 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.437176    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-podinfo\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.437203 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.437191    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-envoy\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.447360 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.447335    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-token\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.447704 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.447682    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmlqt\" (UniqueName: \"kubernetes.io/projected/04b1d5ca-641e-46f4-92ab-9e3c95014445-kube-api-access-bmlqt\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:20.535859 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.535781    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:06:20.535992 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.535856    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr" podUID="cf21565a-4381-4ed0-ad6c-015853c23d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 19:06:20.550079 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.550049    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"
Oct 30 19:06:20.550520 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.550073    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8" event={"ID":"b30f3c4f-5c7a-4766-bc11-7cdb826aef4c","Type":"ContainerDied","Data":"cc78031701cf9cac2f8fccf47d333413b160af25649061f92d225df0bfc8e99e"}
Oct 30 19:06:20.550520 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.550142    4372 scope.go:122] "RemoveContainer" containerID="4478bca04df5854fa8fd4305a190f2c107d8bbc2a370a539dc2060eceb562ddc"
Oct 30 19:06:20.576047 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.576021    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"]
Oct 30 19:06:20.606632 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.606609    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-c9e9de/autores-model-c9e9de-kserve-dbb5fc985-xt7x8"]
Oct 30 19:06:20.703154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:20.701516    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:21.101196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.101060    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"]
Oct 30 19:06:21.113238 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.113189    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:21.113354 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.113296    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:21.113354 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.113345    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:21.536545 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.536451    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:06:21.536730 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.536542    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr" podUID="cf21565a-4381-4ed0-ad6c-015853c23d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 19:06:21.560040 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.559997    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb" event={"ID":"04b1d5ca-641e-46f4-92ab-9e3c95014445","Type":"ContainerStarted","Data":"d28c249c09ebae389128f2c7219de905757350011fa99e0324957f3e604d8607"}
Oct 30 19:06:21.560527 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.560068    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb" event={"ID":"04b1d5ca-641e-46f4-92ab-9e3c95014445","Type":"ContainerStarted","Data":"2e89780370afd8da35aed97aaf50568a394d8a7ad4c7b773d3b3511dbeca02aa"}
Oct 30 19:06:21.613189 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.612768    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb" podStartSLOduration=1.612746104 podStartE2EDuration="1.612746104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:21.598570732 +0000 UTC m=+1995.466989063" watchObservedRunningTime="2026-10-30 19:06:21.612746104 +0000 UTC m=+1995.481164436"
Oct 30 19:06:21.709263 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.709225    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:21.807312 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.807229    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"]
Oct 30 19:06:21.817308 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.814602    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:21.817308 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.817285    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"]
Oct 30 19:06:21.821034 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.821009    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d72beb-openshift-default-dockercfg-drvvh\""
Oct 30 19:06:21.909845 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.909797    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:21.910020 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.909869    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c4e204-f99c-46d4-bbb8-340438118869-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:21.910020 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.909901    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94shs\" (UniqueName: \"kubernetes.io/projected/f4c4e204-f99c-46d4-bbb8-340438118869-kube-api-access-94shs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:21.910020 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.909933    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:21.910159 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.910052    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c4e204-f99c-46d4-bbb8-340438118869-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:21.910159 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.910088    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:21.910417 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.910325    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:21.914203 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.913886    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:21.914203 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:21.913933    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c4e204-f99c-46d4-bbb8-340438118869-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.015272 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.015236    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.015454 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.015280    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c4e204-f99c-46d4-bbb8-340438118869-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.015454 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.015411    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.015583 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.015461    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c4e204-f99c-46d4-bbb8-340438118869-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.015583 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.015493    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94shs\" (UniqueName: \"kubernetes.io/projected/f4c4e204-f99c-46d4-bbb8-340438118869-kube-api-access-94shs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.015583 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.015543    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.015740 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.015651    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.015740 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.015676    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c4e204-f99c-46d4-bbb8-340438118869-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.015740 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.015712    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.015897 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.015759    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.015897 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.015872    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.016024 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.015955    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.016167 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.016110    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.016245 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.016227    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c4e204-f99c-46d4-bbb8-340438118869-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.018284 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.018255    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.018458 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.018442    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c4e204-f99c-46d4-bbb8-340438118869-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.031186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.031165    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94shs\" (UniqueName: \"kubernetes.io/projected/f4c4e204-f99c-46d4-bbb8-340438118869-kube-api-access-94shs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.038293 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.035293    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c4e204-f99c-46d4-bbb8-340438118869-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.136529 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.136488    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:22.326770 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.324767    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30f3c4f-5c7a-4766-bc11-7cdb826aef4c" path="/var/lib/kubelet/pods/b30f3c4f-5c7a-4766-bc11-7cdb826aef4c/volumes"
Oct 30 19:06:22.559231 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.555302    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:22.559231 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.557054    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:22.559231 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.558304    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:06:22.566462 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.565589    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"]
Oct 30 19:06:22.618733 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.618696    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:22.618886 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.618761    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:22.618886 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.618796    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:22.719609 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.717281    4372 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:06:22.719609 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:22.717389    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb" podUID="04b1d5ca-641e-46f4-92ab-9e3c95014445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 19:06:23.609546 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:23.609504    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6" event={"ID":"f4c4e204-f99c-46d4-bbb8-340438118869","Type":"ContainerStarted","Data":"ff09c5b035f95b9146b23e82e1f24179a07446569ed5b3b38968354e8346d366"}
Oct 30 19:06:23.610054 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:23.609587    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6" event={"ID":"f4c4e204-f99c-46d4-bbb8-340438118869","Type":"ContainerStarted","Data":"b5171aedf5072f2fc204fdfcb5300edb90d266f72ea8de535fe7d1b84629a174"}
Oct 30 19:06:23.639229 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:23.639182    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6" podStartSLOduration=2.6391697389999997 podStartE2EDuration="2.639169739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:23.634572794 +0000 UTC m=+1997.502991125" watchObservedRunningTime="2026-10-30 19:06:23.639169739 +0000 UTC m=+1997.507588069"
Oct 30 19:06:23.703647 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:23.703613    4372 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:06:23.703814 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:23.703671    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb" podUID="04b1d5ca-641e-46f4-92ab-9e3c95014445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 19:06:24.137556 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:24.137521    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:24.139305 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:24.139270    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:06:24.139466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:24.139319    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6" podUID="f4c4e204-f99c-46d4-bbb8-340438118869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 19:06:24.715149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:24.714302    4372 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:06:24.715149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:24.714364    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb" podUID="04b1d5ca-641e-46f4-92ab-9e3c95014445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 19:06:25.138239 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:25.138206    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:06:25.138439 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:25.138256    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6" podUID="f4c4e204-f99c-46d4-bbb8-340438118869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 19:06:25.703539 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:25.703501    4372 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:06:25.703827 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:25.703795    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb" podUID="04b1d5ca-641e-46f4-92ab-9e3c95014445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 19:06:26.137559 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:26.137520    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:06:26.138019 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:26.137570    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6" podUID="f4c4e204-f99c-46d4-bbb8-340438118869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 19:06:26.705532 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:26.705494    4372 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:06:26.705820 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:26.705791    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb" podUID="04b1d5ca-641e-46f4-92ab-9e3c95014445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 19:06:27.137710 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:27.137660    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:06:27.138181 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:27.137735    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6" podUID="f4c4e204-f99c-46d4-bbb8-340438118869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 19:06:27.718154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:27.713716    4372 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:06:27.718154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:27.713775    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb" podUID="04b1d5ca-641e-46f4-92ab-9e3c95014445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 19:06:27.823151 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:06:27.821701    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:06:28.137560 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:28.137517    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:06:28.137743 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:28.137603    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6" podUID="f4c4e204-f99c-46d4-bbb8-340438118869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 19:06:29.022200 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:29.017545    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:29.022200 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:29.019208    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:29.022792 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:29.022719    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:29.139188 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:29.139147    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:06:29.139668 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:29.139230    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6" podUID="f4c4e204-f99c-46d4-bbb8-340438118869" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 19:06:30.190319 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:30.190260    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:30.191368 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:30.191309    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:30.203878 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:30.203847    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:06:35.883350 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:35.883302    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:06:36.619998 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:36.619790    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"]
Oct 30 19:06:36.620248 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:36.620050    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb" podUID="04b1d5ca-641e-46f4-92ab-9e3c95014445" containerName="istio-proxy" containerID="cri-o://d28c249c09ebae389128f2c7219de905757350011fa99e0324957f3e604d8607" gracePeriod=30
Oct 30 19:06:41.918526 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:41.917862    4372 generic.go:356] "Generic (PLEG): container finished" podID="04b1d5ca-641e-46f4-92ab-9e3c95014445" containerID="d28c249c09ebae389128f2c7219de905757350011fa99e0324957f3e604d8607" exitCode=0
Oct 30 19:06:41.918526 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:41.918138    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb" event={"ID":"04b1d5ca-641e-46f4-92ab-9e3c95014445","Type":"ContainerDied","Data":"d28c249c09ebae389128f2c7219de905757350011fa99e0324957f3e604d8607"}
Oct 30 19:06:42.062883 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.062855    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:42.157252 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.157208    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-workload-socket\") pod \"04b1d5ca-641e-46f4-92ab-9e3c95014445\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") "
Oct 30 19:06:42.157441 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.157295    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-credential-socket\") pod \"04b1d5ca-641e-46f4-92ab-9e3c95014445\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") "
Oct 30 19:06:42.157441 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.157393    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-token\" (UniqueName: \"kubernetes.io/projected/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-token\") pod \"04b1d5ca-641e-46f4-92ab-9e3c95014445\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") "
Oct 30 19:06:42.157569 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.157456    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b1d5ca-641e-46f4-92ab-9e3c95014445-kube-api-access-bmlqt\" (UniqueName: \"kubernetes.io/projected/04b1d5ca-641e-46f4-92ab-9e3c95014445-kube-api-access-bmlqt\") pod \"04b1d5ca-641e-46f4-92ab-9e3c95014445\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") "
Oct 30 19:06:42.157569 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.157488    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-workload-certs\") pod \"04b1d5ca-641e-46f4-92ab-9e3c95014445\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") "
Oct 30 19:06:42.157569 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.157530    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-envoy\") pod \"04b1d5ca-641e-46f4-92ab-9e3c95014445\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") "
Oct 30 19:06:42.157569 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.157537    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-workload-socket" pod "04b1d5ca-641e-46f4-92ab-9e3c95014445" (UID: "04b1d5ca-641e-46f4-92ab-9e3c95014445"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:42.158204 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.158144    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-podinfo\") pod \"04b1d5ca-641e-46f4-92ab-9e3c95014445\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") "
Oct 30 19:06:42.158204 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.158192    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-workload-certs" pod "04b1d5ca-641e-46f4-92ab-9e3c95014445" (UID: "04b1d5ca-641e-46f4-92ab-9e3c95014445"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:42.158444 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.158224    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-data\") pod \"04b1d5ca-641e-46f4-92ab-9e3c95014445\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") "
Oct 30 19:06:42.158444 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.158317    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-credential-socket" pod "04b1d5ca-641e-46f4-92ab-9e3c95014445" (UID: "04b1d5ca-641e-46f4-92ab-9e3c95014445"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:42.158444 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.158323    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04b1d5ca-641e-46f4-92ab-9e3c95014445-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b1d5ca-641e-46f4-92ab-9e3c95014445-istiod-ca-cert\") pod \"04b1d5ca-641e-46f4-92ab-9e3c95014445\" (UID: \"04b1d5ca-641e-46f4-92ab-9e3c95014445\") "
Oct 30 19:06:42.158631 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.158520    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-data" pod "04b1d5ca-641e-46f4-92ab-9e3c95014445" (UID: "04b1d5ca-641e-46f4-92ab-9e3c95014445"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:42.158895 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.158869    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b1d5ca-641e-46f4-92ab-9e3c95014445-istiod-ca-cert" pod "04b1d5ca-641e-46f4-92ab-9e3c95014445" (UID: "04b1d5ca-641e-46f4-92ab-9e3c95014445"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:42.159291 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.159266    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.159398 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.159296    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b1d5ca-641e-46f4-92ab-9e3c95014445-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.159398 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.159312    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.159398 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.159327    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.159398 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.159341    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.160629 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.160591    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-token" pod "04b1d5ca-641e-46f4-92ab-9e3c95014445" (UID: "04b1d5ca-641e-46f4-92ab-9e3c95014445"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:42.161051 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.161024    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-podinfo" pod "04b1d5ca-641e-46f4-92ab-9e3c95014445" (UID: "04b1d5ca-641e-46f4-92ab-9e3c95014445"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:06:42.161951 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.161923    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b1d5ca-641e-46f4-92ab-9e3c95014445-kube-api-access-bmlqt" pod "04b1d5ca-641e-46f4-92ab-9e3c95014445" (UID: "04b1d5ca-641e-46f4-92ab-9e3c95014445"). InnerVolumeSpecName "kube-api-access-bmlqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:42.162623 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.162591    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-envoy" pod "04b1d5ca-641e-46f4-92ab-9e3c95014445" (UID: "04b1d5ca-641e-46f4-92ab-9e3c95014445"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:42.260481 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.260449    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmlqt\" (UniqueName: \"kubernetes.io/projected/04b1d5ca-641e-46f4-92ab-9e3c95014445-kube-api-access-bmlqt\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.260481 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.260476    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.260481 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.260488    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.260699 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.260496    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b1d5ca-641e-46f4-92ab-9e3c95014445-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:42.926696 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.926660    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb" event={"ID":"04b1d5ca-641e-46f4-92ab-9e3c95014445","Type":"ContainerDied","Data":"2e89780370afd8da35aed97aaf50568a394d8a7ad4c7b773d3b3511dbeca02aa"}
Oct 30 19:06:42.927160 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.926720    4372 scope.go:122] "RemoveContainer" containerID="d28c249c09ebae389128f2c7219de905757350011fa99e0324957f3e604d8607"
Oct 30 19:06:42.927160 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.926673    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"
Oct 30 19:06:42.950894 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.950867    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"]
Oct 30 19:06:42.956802 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:42.956778    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-jxvpb"]
Oct 30 19:06:44.267025 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:44.266990    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04b1d5ca-641e-46f4-92ab-9e3c95014445" path="/var/lib/kubelet/pods/04b1d5ca-641e-46f4-92ab-9e3c95014445/volumes"
Oct 30 19:06:45.371686 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.371655    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"]
Oct 30 19:06:45.372469 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.372451    4372 cpu_manager.go:414] "containerMap: removing container" podUID="04b1d5ca-641e-46f4-92ab-9e3c95014445" containerName="istio-proxy"
Oct 30 19:06:45.372528 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.372471    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b1d5ca-641e-46f4-92ab-9e3c95014445" containerName="istio-proxy"
Oct 30 19:06:45.372596 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.372587    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b1d5ca-641e-46f4-92ab-9e3c95014445" containerName="istio-proxy"
Oct 30 19:06:45.377928 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.377910    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"
Oct 30 19:06:45.391027 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.391005    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-689c23\""
Oct 30 19:06:45.391159 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.391065    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-689c23-dockercfg-xx9q7\""
Oct 30 19:06:45.400435 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.400409    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"]
Oct 30 19:06:45.427789 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.427759    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"]
Oct 30 19:06:45.433758 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.433737    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"
Oct 30 19:06:45.441280 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.441247    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c952c12c-9e03-47a0-b24b-a08f75963f2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8\" (UID: \"c952c12c-9e03-47a0-b24b-a08f75963f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"
Oct 30 19:06:45.441594 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.441566    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l85zb\" (UniqueName: \"kubernetes.io/projected/c952c12c-9e03-47a0-b24b-a08f75963f2d-kube-api-access-l85zb\") pod \"payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8\" (UID: \"c952c12c-9e03-47a0-b24b-a08f75963f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"
Oct 30 19:06:45.441695 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.441575    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"]
Oct 30 19:06:45.573181 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.546370    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l85zb\" (UniqueName: \"kubernetes.io/projected/c952c12c-9e03-47a0-b24b-a08f75963f2d-kube-api-access-l85zb\") pod \"payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8\" (UID: \"c952c12c-9e03-47a0-b24b-a08f75963f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"
Oct 30 19:06:45.573181 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.546534    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c952c12c-9e03-47a0-b24b-a08f75963f2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8\" (UID: \"c952c12c-9e03-47a0-b24b-a08f75963f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"
Oct 30 19:06:45.573181 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.547949    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c952c12c-9e03-47a0-b24b-a08f75963f2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8\" (UID: \"c952c12c-9e03-47a0-b24b-a08f75963f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"
Oct 30 19:06:45.669195 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.655886    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crdmn\" (UniqueName: \"kubernetes.io/projected/8eb9bec3-a8fd-4795-be5b-43f346c31cb9-kube-api-access-crdmn\") pod \"payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx\" (UID: \"8eb9bec3-a8fd-4795-be5b-43f346c31cb9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"
Oct 30 19:06:45.669195 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.655958    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb9bec3-a8fd-4795-be5b-43f346c31cb9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx\" (UID: \"8eb9bec3-a8fd-4795-be5b-43f346c31cb9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"
Oct 30 19:06:45.773187 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.763304    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l85zb\" (UniqueName: \"kubernetes.io/projected/c952c12c-9e03-47a0-b24b-a08f75963f2d-kube-api-access-l85zb\") pod \"payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8\" (UID: \"c952c12c-9e03-47a0-b24b-a08f75963f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"
Oct 30 19:06:45.789141 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.789085    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crdmn\" (UniqueName: \"kubernetes.io/projected/8eb9bec3-a8fd-4795-be5b-43f346c31cb9-kube-api-access-crdmn\") pod \"payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx\" (UID: \"8eb9bec3-a8fd-4795-be5b-43f346c31cb9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"
Oct 30 19:06:45.789475 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.789437    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb9bec3-a8fd-4795-be5b-43f346c31cb9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx\" (UID: \"8eb9bec3-a8fd-4795-be5b-43f346c31cb9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"
Oct 30 19:06:45.790697 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.790673    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb9bec3-a8fd-4795-be5b-43f346c31cb9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx\" (UID: \"8eb9bec3-a8fd-4795-be5b-43f346c31cb9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"
Oct 30 19:06:45.814454 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.812002    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crdmn\" (UniqueName: \"kubernetes.io/projected/8eb9bec3-a8fd-4795-be5b-43f346c31cb9-kube-api-access-crdmn\") pod \"payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx\" (UID: \"8eb9bec3-a8fd-4795-be5b-43f346c31cb9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"
Oct 30 19:06:45.819848 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.819816    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"
Oct 30 19:06:45.993664 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:45.993625    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"
Oct 30 19:06:46.151796 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:46.148070    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"]
Oct 30 19:06:46.153633 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:06:46.153595    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb9bec3_a8fd_4795_be5b_43f346c31cb9.slice/crio-f5f3e3d9c45103a15db3193e76972845470a65b23c649ce1241fb3a2e08d9797 WatchSource:0}: Error finding container f5f3e3d9c45103a15db3193e76972845470a65b23c649ce1241fb3a2e08d9797: Status 404 returned error can't find the container with id f5f3e3d9c45103a15db3193e76972845470a65b23c649ce1241fb3a2e08d9797
Oct 30 19:06:46.377334 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:06:46.377300    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc952c12c_9e03_47a0_b24b_a08f75963f2d.slice/crio-e5e2b99ff85f3ec48ae7a4ae1eb4c1eab217b008b1a88919da93238a19c1ebf8 WatchSource:0}: Error finding container e5e2b99ff85f3ec48ae7a4ae1eb4c1eab217b008b1a88919da93238a19c1ebf8: Status 404 returned error can't find the container with id e5e2b99ff85f3ec48ae7a4ae1eb4c1eab217b008b1a88919da93238a19c1ebf8
Oct 30 19:06:46.377743 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:46.377378    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"]
Oct 30 19:06:47.254170 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:47.234966    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx" event={"ID":"8eb9bec3-a8fd-4795-be5b-43f346c31cb9","Type":"ContainerStarted","Data":"41e0dac994798ed1c6ecb22205cbca697b3dc022378ab3285361d44b65e98b2f"}
Oct 30 19:06:47.254170 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:47.235090    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx" event={"ID":"8eb9bec3-a8fd-4795-be5b-43f346c31cb9","Type":"ContainerStarted","Data":"f5f3e3d9c45103a15db3193e76972845470a65b23c649ce1241fb3a2e08d9797"}
Oct 30 19:06:47.294338 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:47.294278    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"
Oct 30 19:06:47.300362 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:47.299669    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8" event={"ID":"c952c12c-9e03-47a0-b24b-a08f75963f2d","Type":"ContainerStarted","Data":"b241159436f60ea2b4701c265e7069efe7e3378df7f5fe193d950a43167b4449"}
Oct 30 19:06:47.300362 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:47.299723    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8" event={"ID":"c952c12c-9e03-47a0-b24b-a08f75963f2d","Type":"ContainerStarted","Data":"e5e2b99ff85f3ec48ae7a4ae1eb4c1eab217b008b1a88919da93238a19c1ebf8"}
Oct 30 19:06:47.300362 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:47.299760    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"
Oct 30 19:06:47.332618 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:47.332538    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx" podStartSLOduration=2.332481579 podStartE2EDuration="2.332481579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:47.328670119 +0000 UTC m=+2021.197088450" watchObservedRunningTime="2026-10-30 19:06:47.332481579 +0000 UTC m=+2021.200899901"
Oct 30 19:06:47.370398 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:47.370353    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8" podStartSLOduration=2.370338351 podStartE2EDuration="2.370338351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:47.364700368 +0000 UTC m=+2021.233118698" watchObservedRunningTime="2026-10-30 19:06:47.370338351 +0000 UTC m=+2021.238756741"
Oct 30 19:06:49.406719 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:49.401421    4372 generic.go:356] "Generic (PLEG): container finished" podID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerID="04a5f256ae5398fcb41558cdad9e438580b3aaacdd46eb94dd7ec1356d055057" exitCode=0
Oct 30 19:06:49.409964 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:49.407447    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" event={"ID":"9313e8c4-ec9e-4735-b58a-ce18fa0a692b","Type":"ContainerDied","Data":"04a5f256ae5398fcb41558cdad9e438580b3aaacdd46eb94dd7ec1356d055057"}
Oct 30 19:06:49.409964 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:49.408429    4372 scope.go:122] "RemoveContainer" containerID="04a5f256ae5398fcb41558cdad9e438580b3aaacdd46eb94dd7ec1356d055057"
Oct 30 19:06:49.429151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:49.415363    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:49.429538 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:49.429509    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:49.429739 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:49.429717    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:06:50.440334 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:50.440295    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" event={"ID":"9313e8c4-ec9e-4735-b58a-ce18fa0a692b","Type":"ContainerStarted","Data":"69fcbf596531795b3d5bce24f7d5093c9c8b2287f646c3236013fdbd8288a9ba"}
Oct 30 19:06:50.726209 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:50.723783    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:06:50.726209 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:50.725463    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:06:50.726209 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:50.725530    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:06:51.719116 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:51.719076    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:06:51.719661 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:51.719176    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:06:52.719657 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:52.719624    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:06:52.720193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:52.719688    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:06:53.719085 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:53.719048    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:06:53.719337 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:53.719110    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:06:54.511576 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:54.511542    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:06:54.719535 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:54.719497    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:06:54.719815 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:54.719574    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:06:55.719462 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:55.719430    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:06:55.719888 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:55.719479    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:06:56.719721 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:56.719686    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:06:56.720172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:56.719741    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:06:57.696993 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:06:57.696954    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:06:57.724990 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:57.724964    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:06:57.726107 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:57.726086    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:06:58.312549 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:58.312519    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"
Oct 30 19:06:59.450355 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:06:59.450322    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"
Oct 30 19:07:00.302962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.302925    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"]
Oct 30 19:07:00.310470 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.310440    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.322488 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.322458    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"]
Oct 30 19:07:00.356288 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.355762    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"]
Oct 30 19:07:00.361770 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.361744    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.374434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.374406    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"]
Oct 30 19:07:00.393648 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.393616    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616634c4-e125-41b1-82e3-db2556beadb9-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.393846 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.393826    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.394056 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394027    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.394203 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394185    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.394382 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394314    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.394471 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394398    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.394471 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394446    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.394592 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394476    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.394592 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394504    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2284506-257f-4b40-a5c0-69282848c9b6-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.394592 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394559    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616634c4-e125-41b1-82e3-db2556beadb9-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.394592 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394587    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2284506-257f-4b40-a5c0-69282848c9b6-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.394820 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394630    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.394820 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394719    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp5vc\" (UniqueName: \"kubernetes.io/projected/e2284506-257f-4b40-a5c0-69282848c9b6-kube-api-access-bp5vc\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.394820 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394744    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr489\" (UniqueName: \"kubernetes.io/projected/616634c4-e125-41b1-82e3-db2556beadb9-kube-api-access-fr489\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.394820 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394777    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.395027 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394819    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2284506-257f-4b40-a5c0-69282848c9b6-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.395027 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394927    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616634c4-e125-41b1-82e3-db2556beadb9-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.395027 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.394979    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.496809 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.496780    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp5vc\" (UniqueName: \"kubernetes.io/projected/e2284506-257f-4b40-a5c0-69282848c9b6-kube-api-access-bp5vc\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.497331 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.496814    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr489\" (UniqueName: \"kubernetes.io/projected/616634c4-e125-41b1-82e3-db2556beadb9-kube-api-access-fr489\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.497331 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.496843    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.497331 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.496903    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2284506-257f-4b40-a5c0-69282848c9b6-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.497331 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497007    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616634c4-e125-41b1-82e3-db2556beadb9-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.497331 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497052    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.497331 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497174    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616634c4-e125-41b1-82e3-db2556beadb9-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.497331 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497206    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.497331 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497232    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.497331 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497242    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.497331 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497304    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.497949 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497357    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.497949 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497416    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.497949 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497443    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.497949 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497465    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.497949 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497487    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2284506-257f-4b40-a5c0-69282848c9b6-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.497949 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497537    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616634c4-e125-41b1-82e3-db2556beadb9-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.497949 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497566    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2284506-257f-4b40-a5c0-69282848c9b6-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.497949 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497647    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.497949 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497659    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.497949 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497707    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.498531 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.497986    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616634c4-e125-41b1-82e3-db2556beadb9-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.498531 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.498006    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.498531 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.498348    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.498531 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.498379    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.499045 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.498851    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.499115 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.499088    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.499414 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.499389    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2284506-257f-4b40-a5c0-69282848c9b6-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.500606 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.500581    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.500697 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.500591    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2284506-257f-4b40-a5c0-69282848c9b6-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.500697 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.500648    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616634c4-e125-41b1-82e3-db2556beadb9-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.501211 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.501188    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.513967 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.513943    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616634c4-e125-41b1-82e3-db2556beadb9-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.514356 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.514332    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2284506-257f-4b40-a5c0-69282848c9b6-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.515000 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.514980    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr489\" (UniqueName: \"kubernetes.io/projected/616634c4-e125-41b1-82e3-db2556beadb9-kube-api-access-fr489\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.516619 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.516585    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp5vc\" (UniqueName: \"kubernetes.io/projected/e2284506-257f-4b40-a5c0-69282848c9b6-kube-api-access-bp5vc\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.630394 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.630371    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:00.689003 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.687977    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:00.925461 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:07:00.925427    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616634c4_e125_41b1_82e3_db2556beadb9.slice/crio-d2569f001e24f40f9900750f13a4f2f6b71083ef1896ef7ff57a4a20c0ff56c2 WatchSource:0}: Error finding container d2569f001e24f40f9900750f13a4f2f6b71083ef1896ef7ff57a4a20c0ff56c2: Status 404 returned error can't find the container with id d2569f001e24f40f9900750f13a4f2f6b71083ef1896ef7ff57a4a20c0ff56c2
Oct 30 19:07:00.925905 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.925883    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"]
Oct 30 19:07:00.928343 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.928308    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:00.928430 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.928375    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:00.928430 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:00.928404    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:01.041197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:01.040700    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"]
Oct 30 19:07:01.045332 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:07:01.045300    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2284506_257f_4b40_a5c0_69282848c9b6.slice/crio-ced7b7ccaf67bbafeb27bf036451c00b72539deae1d8bb941e64ff1c5d5cae14 WatchSource:0}: Error finding container ced7b7ccaf67bbafeb27bf036451c00b72539deae1d8bb941e64ff1c5d5cae14: Status 404 returned error can't find the container with id ced7b7ccaf67bbafeb27bf036451c00b72539deae1d8bb941e64ff1c5d5cae14
Oct 30 19:07:01.052301 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:01.052264    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:01.052407 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:01.052337    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:01.052407 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:01.052370    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:01.571390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:01.568323    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh" event={"ID":"616634c4-e125-41b1-82e3-db2556beadb9","Type":"ContainerStarted","Data":"5674d90315d1186c624820cc6f9b005bd541025e92883a49611b9422ea03877b"}
Oct 30 19:07:01.571390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:01.568393    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh" event={"ID":"616634c4-e125-41b1-82e3-db2556beadb9","Type":"ContainerStarted","Data":"d2569f001e24f40f9900750f13a4f2f6b71083ef1896ef7ff57a4a20c0ff56c2"}
Oct 30 19:07:01.571390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:01.571030    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh" event={"ID":"e2284506-257f-4b40-a5c0-69282848c9b6","Type":"ContainerStarted","Data":"1fb60d5766ffae739154e091bbc4a5dee22f04ce5a1512d60187983196eede0c"}
Oct 30 19:07:01.571390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:01.571080    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh" event={"ID":"e2284506-257f-4b40-a5c0-69282848c9b6","Type":"ContainerStarted","Data":"ced7b7ccaf67bbafeb27bf036451c00b72539deae1d8bb941e64ff1c5d5cae14"}
Oct 30 19:07:01.614343 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:01.614271    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh" podStartSLOduration=1.614213339 podStartE2EDuration="1.614213339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:01.607785673 +0000 UTC m=+2035.476204004" watchObservedRunningTime="2026-10-30 19:07:01.614213339 +0000 UTC m=+2035.482631670"
Oct 30 19:07:01.633151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:01.632216    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:01.666270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:01.666220    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh" podStartSLOduration=1.666204353 podStartE2EDuration="1.666204353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:01.661533442 +0000 UTC m=+2035.529951775" watchObservedRunningTime="2026-10-30 19:07:01.666204353 +0000 UTC m=+2035.534622754"
Oct 30 19:07:01.695151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:01.692214    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:02.633600 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:02.633560    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 19:07:02.634036 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:02.633631    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh" podUID="616634c4-e125-41b1-82e3-db2556beadb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 19:07:02.720635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:02.720265    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 19:07:02.720635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:02.720326    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh" podUID="e2284506-257f-4b40-a5c0-69282848c9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 19:07:03.633527 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:03.633497    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 19:07:03.633781 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:03.633761    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh" podUID="616634c4-e125-41b1-82e3-db2556beadb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 19:07:03.699547 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:03.699507    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 19:07:03.699823 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:03.699793    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh" podUID="e2284506-257f-4b40-a5c0-69282848c9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 19:07:04.631113 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:04.631073    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 19:07:04.631420 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:04.631390    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh" podUID="616634c4-e125-41b1-82e3-db2556beadb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 19:07:04.697152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:04.696427    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 19:07:04.697152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:04.696480    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh" podUID="e2284506-257f-4b40-a5c0-69282848c9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 19:07:05.631670 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:05.631622    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 19:07:05.631871 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:05.631681    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh" podUID="616634c4-e125-41b1-82e3-db2556beadb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 19:07:05.698151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:05.690462    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 19:07:05.698151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:05.690516    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh" podUID="e2284506-257f-4b40-a5c0-69282848c9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 19:07:06.612151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.611167    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"]
Oct 30 19:07:06.632267 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.628088    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.637062 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.636218    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 19:07:06.637062 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.636297    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh" podUID="616634c4-e125-41b1-82e3-db2556beadb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 19:07:06.644771 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.642198    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"]
Oct 30 19:07:06.734150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.733543    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7c6027-openshift-default-dockercfg-7c5ll\""
Oct 30 19:07:06.734688 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.734544    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 19:07:06.734688 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.734611    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh" podUID="e2284506-257f-4b40-a5c0-69282848c9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 19:07:06.736925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.736895    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-workload-certs\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.737300 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.737276    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4290d2e4-ccae-445f-b184-4509020ddefa-istio-podinfo\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.737642 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.737564    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-credential-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.738115 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.738089    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4290d2e4-ccae-445f-b184-4509020ddefa-istiod-ca-cert\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.738270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.738155    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssvkd\" (UniqueName: \"kubernetes.io/projected/4290d2e4-ccae-445f-b184-4509020ddefa-kube-api-access-ssvkd\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.738270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.738239    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-istio-envoy\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.738700 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.738471    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-workload-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.738882 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.738802    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-istio-data\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.738962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.738938    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4290d2e4-ccae-445f-b184-4509020ddefa-istio-token\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.840954 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.840911    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-workload-certs\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.841912 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.841878    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4290d2e4-ccae-445f-b184-4509020ddefa-istio-podinfo\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.842758 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.842738    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-credential-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.843416 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.843390    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4290d2e4-ccae-445f-b184-4509020ddefa-istiod-ca-cert\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.850156 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.843594    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssvkd\" (UniqueName: \"kubernetes.io/projected/4290d2e4-ccae-445f-b184-4509020ddefa-kube-api-access-ssvkd\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.851099 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.851079    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-istio-envoy\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.851659 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.841756    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-workload-certs\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.852339 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.843304    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-credential-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.852488 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.851013    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4290d2e4-ccae-445f-b184-4509020ddefa-istio-podinfo\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.852691 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.845850    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4290d2e4-ccae-445f-b184-4509020ddefa-istiod-ca-cert\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.852818 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.851904    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-workload-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.852999 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.852284    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-workload-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.853225 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.853205    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-istio-data\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.853418 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.853398    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4290d2e4-ccae-445f-b184-4509020ddefa-istio-token\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.856194 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.855546    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-istio-data\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.861967 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.861904    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssvkd\" (UniqueName: \"kubernetes.io/projected/4290d2e4-ccae-445f-b184-4509020ddefa-kube-api-access-ssvkd\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.882213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.879552    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4290d2e4-ccae-445f-b184-4509020ddefa-istio-token\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.882213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.881310    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-istio-envoy\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:06.966934 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:06.966575    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:07.067317 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.067277    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"]
Oct 30 19:07:07.147631 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.147546    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"]
Oct 30 19:07:07.151650 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.151623    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"]
Oct 30 19:07:07.154352 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.151881    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.159287 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.159257    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.163521 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.163501    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"]
Oct 30 19:07:07.165343 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.165322    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.165547 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.165534    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.165634 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.165622    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.165750 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.165741    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.165811 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.165803    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf56cfc-1bc7-4221-a324-f6b885c80415-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.165900 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.165890    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.165972 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.165962    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvp4w\" (UniqueName: \"kubernetes.io/projected/4bf56cfc-1bc7-4221-a324-f6b885c80415-kube-api-access-hvp4w\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.166228 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.166210    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.166363 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.166345    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.267844 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.267687    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.267844 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.267739    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.267844 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.267804    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.268206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.267861    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df604beb-2e90-4717-a9e5-c6786dfe4473-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.268206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.267898    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.268206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.267953    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.268206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.267975    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.268206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.268002    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.268206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.268025    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppc66\" (UniqueName: \"kubernetes.io/projected/df604beb-2e90-4717-a9e5-c6786dfe4473-kube-api-access-ppc66\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.268206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.268047    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.268206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.268073    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.268206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.268092    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.268206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.268115    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.268206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.268167    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf56cfc-1bc7-4221-a324-f6b885c80415-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.268206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.268204    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.268673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.268225    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvp4w\" (UniqueName: \"kubernetes.io/projected/4bf56cfc-1bc7-4221-a324-f6b885c80415-kube-api-access-hvp4w\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.268673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.268243    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.268673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.268278    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.269521 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.269243    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.269521 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.269299    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.269721 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.269544    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.269721 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.269651    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.270011 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.269989    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf56cfc-1bc7-4221-a324-f6b885c80415-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.270889 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.270870    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.271000 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.270981    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.284930 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.284865    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.287589 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.287543    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvp4w\" (UniqueName: \"kubernetes.io/projected/4bf56cfc-1bc7-4221-a324-f6b885c80415-kube-api-access-hvp4w\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.369782 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.369744    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.369982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.369794    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.369982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.369813    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.369982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.369876    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.369982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.369927    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.370293 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.370069    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.370293 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.370182    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df604beb-2e90-4717-a9e5-c6786dfe4473-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.370635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.370333    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.370635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.370402    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.370635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.370405    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.370635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.370444    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppc66\" (UniqueName: \"kubernetes.io/projected/df604beb-2e90-4717-a9e5-c6786dfe4473-kube-api-access-ppc66\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.370635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.370454    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.371149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.370775    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.371149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.370988    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df604beb-2e90-4717-a9e5-c6786dfe4473-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.372549 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.372518    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.373027 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.373007    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.382864 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.382835    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.384204 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.384179    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppc66\" (UniqueName: \"kubernetes.io/projected/df604beb-2e90-4717-a9e5-c6786dfe4473-kube-api-access-ppc66\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.454009 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:07:07.453968    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4290d2e4_ccae_445f_b184_4509020ddefa.slice/crio-ef75dad68b067c40e35585795c6458786e69b379b752517082cde516fedd11ca WatchSource:0}: Error finding container ef75dad68b067c40e35585795c6458786e69b379b752517082cde516fedd11ca: Status 404 returned error can't find the container with id ef75dad68b067c40e35585795c6458786e69b379b752517082cde516fedd11ca
Oct 30 19:07:07.456776 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.456466    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"]
Oct 30 19:07:07.458184 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.458144    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:07.458973 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.458231    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:07.458973 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.458276    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:07.478757 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.478726    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:07.509673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.509641    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:07.638854 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.637436    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:07.639350 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.639330    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:07.645409 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.645321    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:07:07.687070 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.686623    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46" event={"ID":"4290d2e4-ccae-445f-b184-4509020ddefa","Type":"ContainerStarted","Data":"5ebb28d2fe5028ff8ce1765c3c2d47f37101fc1b630a62e3001b25edea512f11"}
Oct 30 19:07:07.688012 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.687262    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46" event={"ID":"4290d2e4-ccae-445f-b184-4509020ddefa","Type":"ContainerStarted","Data":"ef75dad68b067c40e35585795c6458786e69b379b752517082cde516fedd11ca"}
Oct 30 19:07:07.696355 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.696326    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:07.696793 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.696771    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:07.698168 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.698141    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:07:07.729114 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.723861    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46" podStartSLOduration=1.7238439140000001 podStartE2EDuration="1.723843914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:07.71772224 +0000 UTC m=+2041.586140572" watchObservedRunningTime="2026-10-30 19:07:07.723843914 +0000 UTC m=+2041.592262248"
Oct 30 19:07:07.808259 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.808217    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"]
Oct 30 19:07:07.814508 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.814466    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:07.814640 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.814559    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:07.814640 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.814606    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:07.844205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.844175    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"]
Oct 30 19:07:07.849945 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.849811    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:07.849945 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.849892    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:07.849945 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.849939    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:08.006223 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:07.974610    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:08.755823 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:08.755778    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw" event={"ID":"4bf56cfc-1bc7-4221-a324-f6b885c80415","Type":"ContainerStarted","Data":"716dd14a1fd586c8d7a019b0c2d3e3343a724e986db15605f8edb15066bfc54a"}
Oct 30 19:07:08.756019 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:08.755883    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw" event={"ID":"4bf56cfc-1bc7-4221-a324-f6b885c80415","Type":"ContainerStarted","Data":"23ee48a6c923f66993d9552acce656d06606ce7f62a43e1b2d5a67b7a04a54b8"}
Oct 30 19:07:08.759445 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:08.759409    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq" event={"ID":"df604beb-2e90-4717-a9e5-c6786dfe4473","Type":"ContainerStarted","Data":"87ebf638102aa4f870231bfb79747e09bfb6de6b44c4ca895f8a8a6eb6856ddf"}
Oct 30 19:07:08.759636 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:08.759458    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq" event={"ID":"df604beb-2e90-4717-a9e5-c6786dfe4473","Type":"ContainerStarted","Data":"715a0f345fa5051391673f7bd3dda6c86388132e21fc2e1d05f9b5e15a152a5d"}
Oct 30 19:07:08.801149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:08.800695    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw" podStartSLOduration=2.80067188 podStartE2EDuration="2.80067188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:08.789055086 +0000 UTC m=+2042.657473416" watchObservedRunningTime="2026-10-30 19:07:08.80067188 +0000 UTC m=+2042.669090211"
Oct 30 19:07:08.833153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:08.832712    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq" podStartSLOduration=2.832694895 podStartE2EDuration="2.832694895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:08.831955674 +0000 UTC m=+2042.700374007" watchObservedRunningTime="2026-10-30 19:07:08.832694895 +0000 UTC m=+2042.701113227"
Oct 30 19:07:08.969136 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:08.969084    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:07:08.969316 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:08.969248    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46" podUID="4290d2e4-ccae-445f-b184-4509020ddefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 19:07:09.482236 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:09.480470    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:09.510170 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:09.499664    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:07:09.510170 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:09.499730    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw" podUID="4bf56cfc-1bc7-4221-a324-f6b885c80415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 19:07:09.523164 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:09.511671    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:09.523164 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:09.515247    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:07:09.523164 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:09.515307    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq" podUID="df604beb-2e90-4717-a9e5-c6786dfe4473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 19:07:09.967776 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:09.967715    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:07:09.968450 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:09.968414    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46" podUID="4290d2e4-ccae-445f-b184-4509020ddefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 19:07:10.487164 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:10.485468    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:07:10.496270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:10.496231    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw" podUID="4bf56cfc-1bc7-4221-a324-f6b885c80415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 19:07:10.519864 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:10.519721    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:07:10.520108 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:10.520083    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq" podUID="df604beb-2e90-4717-a9e5-c6786dfe4473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 19:07:10.968269 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:10.968234    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:07:10.969025 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:10.968994    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46" podUID="4290d2e4-ccae-445f-b184-4509020ddefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 19:07:11.488822 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:11.488775    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:07:11.489097 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:11.489062    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw" podUID="4bf56cfc-1bc7-4221-a324-f6b885c80415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 19:07:11.517651 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:11.517617    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:07:11.517926 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:11.517898    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq" podUID="df604beb-2e90-4717-a9e5-c6786dfe4473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 19:07:11.966970 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:11.966932    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:07:11.967148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:11.966990    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46" podUID="4290d2e4-ccae-445f-b184-4509020ddefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 19:07:12.483153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:12.482534    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:07:12.483153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:12.482600    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw" podUID="4bf56cfc-1bc7-4221-a324-f6b885c80415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 19:07:12.531296 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:12.531256    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:07:12.531588 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:12.531560    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq" podUID="df604beb-2e90-4717-a9e5-c6786dfe4473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 19:07:12.968181 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:12.968146    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:07:12.968449 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:12.968424    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46" podUID="4290d2e4-ccae-445f-b184-4509020ddefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 19:07:13.486165 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:13.485615    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:07:13.486165 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:13.485676    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw" podUID="4bf56cfc-1bc7-4221-a324-f6b885c80415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 19:07:13.516152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:13.513728    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:07:13.516152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:13.513787    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq" podUID="df604beb-2e90-4717-a9e5-c6786dfe4473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 19:07:13.967711 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:13.967663    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:07:13.968033 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:13.967997    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46" podUID="4290d2e4-ccae-445f-b184-4509020ddefa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 19:07:14.496152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:14.492254    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:07:14.496152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:14.492347    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw" podUID="4bf56cfc-1bc7-4221-a324-f6b885c80415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 19:07:14.521186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:14.521007    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:07:14.521186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:14.521064    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq" podUID="df604beb-2e90-4717-a9e5-c6786dfe4473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 19:07:14.974870 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:14.974836    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:14.975498 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:14.975471    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:14.976366 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:14.976341    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:07:15.485479 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:15.485445    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:15.485922 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:15.485894    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:15.486707 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:15.486683    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:07:15.549024 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:15.548946    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:15.550939 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:15.550913    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:15.552523 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:15.552498    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:07:27.698162 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:07:27.698090    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:07:33.671774 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:33.671731    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:07:53.394900 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:53.394864    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"]
Oct 30 19:07:54.210535 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.210500    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"]
Oct 30 19:07:54.217025 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.217001    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.227982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.227954    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-689c23-openshift-default-dockercfg-ng4kn\""
Oct 30 19:07:54.234930 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.234909    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"]
Oct 30 19:07:54.334229 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.334186    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-data\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.334430 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.334296    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-workload-certs\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.334430 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.334377    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-token\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.334430 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.334412    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-envoy\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.334640 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.334461    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-podinfo\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.334640 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.334616    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-workload-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.334774 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.334647    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2kgf\" (UniqueName: \"kubernetes.io/projected/fbcd714e-b581-4142-ae56-05bf2a33d52b-kube-api-access-g2kgf\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.334855 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.334823    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-credential-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.334973 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.334951    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbcd714e-b581-4142-ae56-05bf2a33d52b-istiod-ca-cert\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.436655 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.436623    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-token\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.437073 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.436670    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-envoy\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.437073 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.436697    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-podinfo\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.437073 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.436740    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-workload-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.437073 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.436773    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2kgf\" (UniqueName: \"kubernetes.io/projected/fbcd714e-b581-4142-ae56-05bf2a33d52b-kube-api-access-g2kgf\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.437073 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.436833    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-credential-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.437073 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.436915    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbcd714e-b581-4142-ae56-05bf2a33d52b-istiod-ca-cert\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.437073 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.436998    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-data\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.437498 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.437090    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-workload-certs\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.437498 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.437234    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-workload-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.437498 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.437312    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-credential-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.437612 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.437521    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-workload-certs\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.437612 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.437510    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-data\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.437992 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.437968    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbcd714e-b581-4142-ae56-05bf2a33d52b-istiod-ca-cert\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.439404 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.439383    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-envoy\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.439646 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.439624    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-podinfo\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.449861 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.449836    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2kgf\" (UniqueName: \"kubernetes.io/projected/fbcd714e-b581-4142-ae56-05bf2a33d52b-kube-api-access-g2kgf\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.450206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.450185    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-token\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.533207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.530428    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"]
Oct 30 19:07:54.558331 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.556667    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:54.893114 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.893087    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"]
Oct 30 19:07:54.897236 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.897197    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:54.897327 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.897289    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:54.897368 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:54.897333    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:07:55.447418 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:55.447309    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb" event={"ID":"fbcd714e-b581-4142-ae56-05bf2a33d52b","Type":"ContainerStarted","Data":"c16eec50949089e15a1cd73b971848a5576c218980eca67630c86732b0829293"}
Oct 30 19:07:55.448189 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:55.448118    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb" event={"ID":"fbcd714e-b581-4142-ae56-05bf2a33d52b","Type":"ContainerStarted","Data":"392a42698e3b8b80fc421ebb5a9a3264c4b88d7d901d14c2aa7fe202feabe18b"}
Oct 30 19:07:55.479066 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:55.479021    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb" podStartSLOduration=1.479008245 podStartE2EDuration="1.479008245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:55.473316847 +0000 UTC m=+2089.341735177" watchObservedRunningTime="2026-10-30 19:07:55.479008245 +0000 UTC m=+2089.347426575"
Oct 30 19:07:55.559134 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:55.559107    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:07:55.560294 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:55.560273    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 19:07:55.560358 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:55.560318    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb" podUID="fbcd714e-b581-4142-ae56-05bf2a33d52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 19:07:56.581147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:56.566697    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 19:07:56.581815 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:56.581776    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb" podUID="fbcd714e-b581-4142-ae56-05bf2a33d52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 19:07:57.558553 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:57.558523    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 19:07:57.558789 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:57.558770    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb" podUID="fbcd714e-b581-4142-ae56-05bf2a33d52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 19:07:57.725036 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:07:57.724997    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:07:58.402388 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:58.402203    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q" podUID="82ae5b4b-4033-4fe5-93ab-cbdd22757d08" containerName="payload-processing" containerID="cri-o://7246634a8465a1923f6d75084aa6481c2557acb27b906aaecdfe3ed28eda73ac" gracePeriod=25
Oct 30 19:07:58.527395 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:58.527343    4372 generic.go:356] "Generic (PLEG): container finished" podID="82ae5b4b-4033-4fe5-93ab-cbdd22757d08" containerID="7246634a8465a1923f6d75084aa6481c2557acb27b906aaecdfe3ed28eda73ac" exitCode=0
Oct 30 19:07:58.527582 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:58.527421    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q" event={"ID":"82ae5b4b-4033-4fe5-93ab-cbdd22757d08","Type":"ContainerDied","Data":"7246634a8465a1923f6d75084aa6481c2557acb27b906aaecdfe3ed28eda73ac"}
Oct 30 19:07:58.568154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:58.564251    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 19:07:58.568154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:58.564329    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb" podUID="fbcd714e-b581-4142-ae56-05bf2a33d52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 19:07:58.852373 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:58.852338    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:07:59.018339 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.018252    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-routing-overlay-volume\") pod \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") "
Oct 30 19:07:59.018531 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.018480    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-kube-api-access-29zs4\" (UniqueName: \"kubernetes.io/projected/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-kube-api-access-29zs4\") pod \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") "
Oct 30 19:07:59.018713 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.018689    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-routing-overlay-volume" pod "82ae5b4b-4033-4fe5-93ab-cbdd22757d08" (UID: "82ae5b4b-4033-4fe5-93ab-cbdd22757d08"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:59.018805 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.018715    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-provider-credentials\" (UniqueName: \"kubernetes.io/projected/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-provider-credentials\") pod \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") "
Oct 30 19:07:59.018886 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.018823    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-plugins-config-volume\") pod \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\" (UID: \"82ae5b4b-4033-4fe5-93ab-cbdd22757d08\") "
Oct 30 19:07:59.019373 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.019334    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-plugins-config-volume" pod "82ae5b4b-4033-4fe5-93ab-cbdd22757d08" (UID: "82ae5b4b-4033-4fe5-93ab-cbdd22757d08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:59.019648 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.019570    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:59.019648 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.019593    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:59.021497 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.021475    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-kube-api-access-29zs4" pod "82ae5b4b-4033-4fe5-93ab-cbdd22757d08" (UID: "82ae5b4b-4033-4fe5-93ab-cbdd22757d08"). InnerVolumeSpecName "kube-api-access-29zs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:59.021586 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.021507    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-provider-credentials" pod "82ae5b4b-4033-4fe5-93ab-cbdd22757d08" (UID: "82ae5b4b-4033-4fe5-93ab-cbdd22757d08"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:59.120898 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.120866    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:59.120898 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.120897    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29zs4\" (UniqueName: \"kubernetes.io/projected/82ae5b4b-4033-4fe5-93ab-cbdd22757d08-kube-api-access-29zs4\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:59.563581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.562223    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk" podUID="de7e1b40-890b-4297-b605-e948a6170c7e" containerName="payload-processing" containerID="cri-o://d54edf8ac261a51c0d7a9e0a2a25f3bdb71f46e8a0346bbb48ec766d138ea9a0" gracePeriod=25
Oct 30 19:07:59.563581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.562645    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 19:07:59.563581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.562701    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb" podUID="fbcd714e-b581-4142-ae56-05bf2a33d52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 19:07:59.616738 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.614711    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q" event={"ID":"82ae5b4b-4033-4fe5-93ab-cbdd22757d08","Type":"ContainerDied","Data":"42498adee09fe496e2e9a2f4570e9916b6d6905d322cab0a496a9e8ab4fc7a3f"}
Oct 30 19:07:59.616738 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.614786    4372 scope.go:122] "RemoveContainer" containerID="7246634a8465a1923f6d75084aa6481c2557acb27b906aaecdfe3ed28eda73ac"
Oct 30 19:07:59.616738 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.614998    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"
Oct 30 19:07:59.987154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:07:59.975301    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"]
Oct 30 19:08:00.011847 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:00.011800    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfdgr7q"]
Oct 30 19:08:00.312192 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:00.312092    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ae5b4b-4033-4fe5-93ab-cbdd22757d08" path="/var/lib/kubelet/pods/82ae5b4b-4033-4fe5-93ab-cbdd22757d08/volumes"
Oct 30 19:08:00.521221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:00.505274    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:00.559707 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:00.559285    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 19:08:00.559707 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:00.559343    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb" podUID="fbcd714e-b581-4142-ae56-05bf2a33d52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 19:08:00.864393 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:00.864033    4372 generic.go:356] "Generic (PLEG): container finished" podID="de7e1b40-890b-4297-b605-e948a6170c7e" containerID="d54edf8ac261a51c0d7a9e0a2a25f3bdb71f46e8a0346bbb48ec766d138ea9a0" exitCode=0
Oct 30 19:08:00.864393 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:00.864116    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk" event={"ID":"de7e1b40-890b-4297-b605-e948a6170c7e","Type":"ContainerDied","Data":"d54edf8ac261a51c0d7a9e0a2a25f3bdb71f46e8a0346bbb48ec766d138ea9a0"}
Oct 30 19:08:01.101149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.092531    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:08:01.209261 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.207645    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7e1b40-890b-4297-b605-e948a6170c7e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de7e1b40-890b-4297-b605-e948a6170c7e-routing-overlay-volume\") pod \"de7e1b40-890b-4297-b605-e948a6170c7e\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") "
Oct 30 19:08:01.209261 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.207802    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7e1b40-890b-4297-b605-e948a6170c7e-kube-api-access-5mrmq\" (UniqueName: \"kubernetes.io/projected/de7e1b40-890b-4297-b605-e948a6170c7e-kube-api-access-5mrmq\") pod \"de7e1b40-890b-4297-b605-e948a6170c7e\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") "
Oct 30 19:08:01.209261 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.207983    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7e1b40-890b-4297-b605-e948a6170c7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de7e1b40-890b-4297-b605-e948a6170c7e-plugins-config-volume\") pod \"de7e1b40-890b-4297-b605-e948a6170c7e\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") "
Oct 30 19:08:01.209261 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.208182    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7e1b40-890b-4297-b605-e948a6170c7e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/de7e1b40-890b-4297-b605-e948a6170c7e-provider-credentials\") pod \"de7e1b40-890b-4297-b605-e948a6170c7e\" (UID: \"de7e1b40-890b-4297-b605-e948a6170c7e\") "
Oct 30 19:08:01.210313 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.210279    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7e1b40-890b-4297-b605-e948a6170c7e-routing-overlay-volume" pod "de7e1b40-890b-4297-b605-e948a6170c7e" (UID: "de7e1b40-890b-4297-b605-e948a6170c7e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:01.210532 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.210509    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7e1b40-890b-4297-b605-e948a6170c7e-plugins-config-volume" pod "de7e1b40-890b-4297-b605-e948a6170c7e" (UID: "de7e1b40-890b-4297-b605-e948a6170c7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:01.211771 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.211735    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7e1b40-890b-4297-b605-e948a6170c7e-kube-api-access-5mrmq" pod "de7e1b40-890b-4297-b605-e948a6170c7e" (UID: "de7e1b40-890b-4297-b605-e948a6170c7e"). InnerVolumeSpecName "kube-api-access-5mrmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:01.211914 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.211808    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de7e1b40-890b-4297-b605-e948a6170c7e-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:01.211914 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.211830    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de7e1b40-890b-4297-b605-e948a6170c7e-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:01.214526 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.214490    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7e1b40-890b-4297-b605-e948a6170c7e-provider-credentials" pod "de7e1b40-890b-4297-b605-e948a6170c7e" (UID: "de7e1b40-890b-4297-b605-e948a6170c7e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:01.313736 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.313692    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de7e1b40-890b-4297-b605-e948a6170c7e-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:01.313736 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.313729    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mrmq\" (UniqueName: \"kubernetes.io/projected/de7e1b40-890b-4297-b605-e948a6170c7e-kube-api-access-5mrmq\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:01.558678 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.558592    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 19:08:01.558678 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.558645    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb" podUID="fbcd714e-b581-4142-ae56-05bf2a33d52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 19:08:01.874794 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.874766    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"
Oct 30 19:08:01.874962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.874795    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk" event={"ID":"de7e1b40-890b-4297-b605-e948a6170c7e","Type":"ContainerDied","Data":"4b534ad9fa3d0dcc87e3942aad25a0e71c3dacc31f51f00179440e8768f146dc"}
Oct 30 19:08:01.874962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.874874    4372 scope.go:122] "RemoveContainer" containerID="d54edf8ac261a51c0d7a9e0a2a25f3bdb71f46e8a0346bbb48ec766d138ea9a0"
Oct 30 19:08:01.991208 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:01.991092    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"]
Oct 30 19:08:02.042553 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:02.042516    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfbkhvk"]
Oct 30 19:08:02.269007 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:02.268927    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7e1b40-890b-4297-b605-e948a6170c7e" path="/var/lib/kubelet/pods/de7e1b40-890b-4297-b605-e948a6170c7e/volumes"
Oct 30 19:08:02.564296 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:02.564220    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:08:02.564639 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:02.564609    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:08:02.565098 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:02.565082    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:08:27.701433 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:08:27.701336    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:08:31.099751 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:31.099715    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:08:33.891344 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:33.890240    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:39.396153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.396064    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"]
Oct 30 19:08:39.417163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.397342    4372 cpu_manager.go:414] "containerMap: removing container" podUID="de7e1b40-890b-4297-b605-e948a6170c7e" containerName="payload-processing"
Oct 30 19:08:39.417163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.397368    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7e1b40-890b-4297-b605-e948a6170c7e" containerName="payload-processing"
Oct 30 19:08:39.417163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.397451    4372 cpu_manager.go:414] "containerMap: removing container" podUID="82ae5b4b-4033-4fe5-93ab-cbdd22757d08" containerName="payload-processing"
Oct 30 19:08:39.417163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.397460    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ae5b4b-4033-4fe5-93ab-cbdd22757d08" containerName="payload-processing"
Oct 30 19:08:39.417163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.397676    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ae5b4b-4033-4fe5-93ab-cbdd22757d08" containerName="payload-processing"
Oct 30 19:08:39.417163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.397695    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7e1b40-890b-4297-b605-e948a6170c7e" containerName="payload-processing"
Oct 30 19:08:39.417163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.415182    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.440649 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.440600    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"]
Oct 30 19:08:39.530884 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.530845    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-workload-certs\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.531758 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.531730    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-credential-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.531976 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.531957    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istiod-ca-cert\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.532222 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.532204    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-data\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.532506 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.532489    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-podinfo\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.532642 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.532626    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-token\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.532750 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.532729    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgzr8\" (UniqueName: \"kubernetes.io/projected/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-kube-api-access-xgzr8\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.532904 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.532760    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-envoy\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.532904 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.532868    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-workload-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.665206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.635783    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-credential-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.680295 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.638581    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-credential-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.681296 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.681223    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istiod-ca-cert\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.681455 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.681324    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-data\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.681455 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.681431    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-podinfo\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.681574 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.681482    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-token\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.681574 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.681507    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgzr8\" (UniqueName: \"kubernetes.io/projected/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-kube-api-access-xgzr8\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.681574 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.681535    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-envoy\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.681718 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.681662    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-workload-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.681810 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.681791    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-workload-certs\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.682406 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.682377    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-workload-certs\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.696251 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.683113    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istiod-ca-cert\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.696251 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.683406    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-data\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.699366 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.699329    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-workload-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.753150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.747924    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-podinfo\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.783293 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.779783    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-token\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.783293 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.782935    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-envoy\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:39.808148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:39.802691    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgzr8\" (UniqueName: \"kubernetes.io/projected/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-kube-api-access-xgzr8\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:40.057366 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:40.057290    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:40.552435 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:08:40.552397    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5bee02_c4e2_4043_a111_f5d3d6a3cc4c.slice/crio-a4514d1e4c4cc2984d70aa6d3863794a40f29ac8c3cc2cc46ba6f0a359e61c01 WatchSource:0}: Error finding container a4514d1e4c4cc2984d70aa6d3863794a40f29ac8c3cc2cc46ba6f0a359e61c01: Status 404 returned error can't find the container with id a4514d1e4c4cc2984d70aa6d3863794a40f29ac8c3cc2cc46ba6f0a359e61c01
Oct 30 19:08:40.554397 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:40.554360    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"]
Oct 30 19:08:40.556169 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:40.556112    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:40.556266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:40.556222    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:40.556266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:40.556253    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:41.496906 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:41.496863    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm" event={"ID":"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c","Type":"ContainerStarted","Data":"cf9d13091e025a584ec60732fd114bd0e7252ca26a0649c0ad5c7a8dc37469fa"}
Oct 30 19:08:41.497095 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:41.496933    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm" event={"ID":"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c","Type":"ContainerStarted","Data":"a4514d1e4c4cc2984d70aa6d3863794a40f29ac8c3cc2cc46ba6f0a359e61c01"}
Oct 30 19:08:41.524803 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:41.524738    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm" podStartSLOduration=2.524725478 podStartE2EDuration="2.524725478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:41.522857095 +0000 UTC m=+2135.391275426" watchObservedRunningTime="2026-10-30 19:08:41.524725478 +0000 UTC m=+2135.393143809"
Oct 30 19:08:42.062052 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:42.061824    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:42.063909 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:42.063880    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 19:08:42.064159 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:42.064112    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm" podUID="6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 19:08:43.057791 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:43.057759    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 19:08:43.057973 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:43.057824    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm" podUID="6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 19:08:44.058199 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:44.058164    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 19:08:44.058725 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:44.058236    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm" podUID="6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 19:08:45.065907 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:45.059701    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 19:08:45.065907 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:45.059771    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm" podUID="6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 19:08:46.063558 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:46.063518    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 19:08:46.063863 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:46.063833    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm" podUID="6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 19:08:47.057830 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:47.057790    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 19:08:47.058358 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:47.057863    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm" podUID="6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 19:08:48.064148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.058448    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 19:08:48.064148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.058539    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm" podUID="6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 19:08:48.537503 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.537469    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"]
Oct 30 19:08:48.549150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.546617    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"]
Oct 30 19:08:48.549150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.546744    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:08:48.554558 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.554531    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"agc-cred-ce5ac7999d53\""
Oct 30 19:08:48.556354 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.556323    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"agc-cred-a3eaf63397c9\""
Oct 30 19:08:48.556518 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.556393    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"agc-cred-394aab6f23b5\""
Oct 30 19:08:48.557192 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.557168    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"kube-root-ca.crt\""
Oct 30 19:08:48.557315 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.557263    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"payload-processing-external-model-e2e--0d1ddd5a-dockercfg-7twb9\""
Oct 30 19:08:48.557315 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.557167    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4e94aa\""
Oct 30 19:08:48.557449 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.557348    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"routing-overlay\""
Oct 30 19:08:48.557515 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.557473    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"openshift-service-ca.crt\""
Oct 30 19:08:48.650218 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.650184    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/57458a83-1bee-4116-8b6e-5f511cc9d771-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:08:48.650392 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.650336    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/57458a83-1bee-4116-8b6e-5f511cc9d771-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:08:48.650665 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.650646    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57458a83-1bee-4116-8b6e-5f511cc9d771-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:08:48.650760 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.650747    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45dbq\" (UniqueName: \"kubernetes.io/projected/57458a83-1bee-4116-8b6e-5f511cc9d771-kube-api-access-45dbq\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:08:48.752468 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.752440    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45dbq\" (UniqueName: \"kubernetes.io/projected/57458a83-1bee-4116-8b6e-5f511cc9d771-kube-api-access-45dbq\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:08:48.752677 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.752659    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/57458a83-1bee-4116-8b6e-5f511cc9d771-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:08:48.752771 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.752757    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/57458a83-1bee-4116-8b6e-5f511cc9d771-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:08:48.752840 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.752828    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57458a83-1bee-4116-8b6e-5f511cc9d771-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:08:48.753587 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.753557    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/57458a83-1bee-4116-8b6e-5f511cc9d771-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:08:48.753668 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.753639    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57458a83-1bee-4116-8b6e-5f511cc9d771-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:08:48.755442 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.755424    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/57458a83-1bee-4116-8b6e-5f511cc9d771-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:08:48.765350 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.765327    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45dbq\" (UniqueName: \"kubernetes.io/projected/57458a83-1bee-4116-8b6e-5f511cc9d771-kube-api-access-45dbq\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:08:48.871753 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:48.871729    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:08:49.120151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:49.119203    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:49.121196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:49.121170    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:49.133326 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:49.132482    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:08:49.170412 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:49.170388    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"]
Oct 30 19:08:49.795917 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:49.795878    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj" event={"ID":"57458a83-1bee-4116-8b6e-5f511cc9d771","Type":"ContainerStarted","Data":"fc350606c52f1294c823a4af9bdd8c1a154ace6a8e1d75ad84b46727b33d7ff1"}
Oct 30 19:08:49.796114 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:49.795930    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj" event={"ID":"57458a83-1bee-4116-8b6e-5f511cc9d771","Type":"ContainerStarted","Data":"efeef7dce6f96f9d36c5f7a031a0bb8c7bf99b34afde7d44cd0bf99402ba1b8e"}
Oct 30 19:08:49.796114 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:49.795974    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:08:49.816807 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:49.816767    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj" podStartSLOduration=1.816755905 podStartE2EDuration="1.816755905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:49.81641472 +0000 UTC m=+2143.684833051" watchObservedRunningTime="2026-10-30 19:08:49.816755905 +0000 UTC m=+2143.685174234"
Oct 30 19:08:51.790973 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:51.790942    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"]
Oct 30 19:08:51.796604 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:51.796582    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:51.830152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:51.830107    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"]
Oct 30 19:08:51.929893 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:51.929858    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-workload-certs\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:51.929893 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:51.929895    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k24j9\" (UniqueName: \"kubernetes.io/projected/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-kube-api-access-k24j9\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:51.930164 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:51.929921    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-credential-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:51.930164 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:51.930080    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-data\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:51.930164 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:51.930139    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-envoy\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:51.930330 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:51.930179    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istiod-ca-cert\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:51.930330 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:51.930202    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-token\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:51.930436 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:51.930330    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-workload-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:51.930436 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:51.930408    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-podinfo\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.031657 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.031626    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-workload-certs\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.031841 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.031661    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k24j9\" (UniqueName: \"kubernetes.io/projected/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-kube-api-access-k24j9\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.031841 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.031691    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-credential-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.031841 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.031729    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-data\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.031841 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.031747    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-envoy\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.031841 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.031769    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istiod-ca-cert\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.031841 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.031799    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-token\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.032312 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.031909    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-workload-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.032312 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.031963    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-podinfo\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.032312 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.032095    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-workload-certs\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.032312 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.032186    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-data\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.032312 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.032235    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-credential-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.032496 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.032386    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-workload-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.032834 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.032809    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istiod-ca-cert\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.034351 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.034328    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-envoy\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.034676 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.034655    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-podinfo\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.048314 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.048262    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-token\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.048397 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.048374    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k24j9\" (UniqueName: \"kubernetes.io/projected/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-kube-api-access-k24j9\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.123930 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.123901    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:52.469780 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.469742    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"]
Oct 30 19:08:52.479574 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:08:52.479540    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac6b8fc_c95d_43a7_b18d_2c4953b9cda4.slice/crio-3991e712e45248d097092a192cc4c27997103facc250c3f3b3f5da021a012d40 WatchSource:0}: Error finding container 3991e712e45248d097092a192cc4c27997103facc250c3f3b3f5da021a012d40: Status 404 returned error can't find the container with id 3991e712e45248d097092a192cc4c27997103facc250c3f3b3f5da021a012d40
Oct 30 19:08:52.483516 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.483493    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"]
Oct 30 19:08:52.483638 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.483603    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.483712 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.483681    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:52.483778 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.483754    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:52.483877 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.483797    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:52.501691 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.501661    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"]
Oct 30 19:08:52.502623 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.502007    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e77b1d-openshift-default-dockercfg-9c6tb\""
Oct 30 19:08:52.550182 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.549873    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-data\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.550182 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.549977    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-credential-socket\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.550182 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.550010    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-token\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.550611 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.550570    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-podinfo\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.550736 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.550680    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3a7f70-606a-40b1-b541-e279a8438a32-istiod-ca-cert\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.550795 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.550761    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-envoy\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.551530 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.551148    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-workload-certs\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.551530 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.551188    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl9hr\" (UniqueName: \"kubernetes.io/projected/2e3a7f70-606a-40b1-b541-e279a8438a32-kube-api-access-zl9hr\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.551530 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.551218    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-workload-socket\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.652630 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.652598    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-workload-certs\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.652819 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.652636    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl9hr\" (UniqueName: \"kubernetes.io/projected/2e3a7f70-606a-40b1-b541-e279a8438a32-kube-api-access-zl9hr\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.652819 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.652660    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-workload-socket\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.652956 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.652818    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-data\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.652956 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.652880    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-credential-socket\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.652956 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.652903    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-token\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.653110 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.652963    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-podinfo\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.653110 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.653029    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3a7f70-606a-40b1-b541-e279a8438a32-istiod-ca-cert\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.653110 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.653099    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-envoy\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.653360 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.653099    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-workload-certs\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.653360 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.653165    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-workload-socket\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.653476 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.653438    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-data\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.653831 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.653671    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-credential-socket\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.653959 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.653924    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3a7f70-606a-40b1-b541-e279a8438a32-istiod-ca-cert\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.655677 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.655657    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-envoy\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.655770 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.655701    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-podinfo\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.669899 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.669874    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-token\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.671495 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.671477    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl9hr\" (UniqueName: \"kubernetes.io/projected/2e3a7f70-606a-40b1-b541-e279a8438a32-kube-api-access-zl9hr\") pod \"e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") " pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.828658 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.828615    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:52.836606 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.836568    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs" event={"ID":"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4","Type":"ContainerStarted","Data":"f717a6ce2d278645d05efe8bc49cf7d3ba901a90eb3ffc4d183fa1c648e93ef0"}
Oct 30 19:08:52.836760 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.836626    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs" event={"ID":"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4","Type":"ContainerStarted","Data":"3991e712e45248d097092a192cc4c27997103facc250c3f3b3f5da021a012d40"}
Oct 30 19:08:52.886814 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:52.886757    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs" podStartSLOduration=1.886744411 podStartE2EDuration="1.886744411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:52.881830615 +0000 UTC m=+2146.750248946" watchObservedRunningTime="2026-10-30 19:08:52.886744411 +0000 UTC m=+2146.755162741"
Oct 30 19:08:53.124100 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:53.124034    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:08:53.125276 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:53.125245    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 19:08:53.125413 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:53.125295    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs" podUID="dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 19:08:53.157006 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:08:53.156839    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3a7f70_606a_40b1_b541_e279a8438a32.slice/crio-b6e6826e9b160ed219d377ffed0be8dcfce6a085ebdf059a19f4047f0bb3eaf0 WatchSource:0}: Error finding container b6e6826e9b160ed219d377ffed0be8dcfce6a085ebdf059a19f4047f0bb3eaf0: Status 404 returned error can't find the container with id b6e6826e9b160ed219d377ffed0be8dcfce6a085ebdf059a19f4047f0bb3eaf0
Oct 30 19:08:53.158365 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:53.158252    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"]
Oct 30 19:08:53.160887 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:53.160852    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:53.161000 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:53.160918    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:53.161000 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:53.160948    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:08:53.847611 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:53.847576    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl" event={"ID":"2e3a7f70-606a-40b1-b541-e279a8438a32","Type":"ContainerStarted","Data":"b6bbfee57bbc93240a048de9058cee48dad0b941a84c815ca05efae66bc6457d"}
Oct 30 19:08:53.848054 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:53.847651    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl" event={"ID":"2e3a7f70-606a-40b1-b541-e279a8438a32","Type":"ContainerStarted","Data":"b6e6826e9b160ed219d377ffed0be8dcfce6a085ebdf059a19f4047f0bb3eaf0"}
Oct 30 19:08:53.875827 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:53.875780    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl" podStartSLOduration=1.875767235 podStartE2EDuration="1.875767235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:53.87509961 +0000 UTC m=+2147.743517942" watchObservedRunningTime="2026-10-30 19:08:53.875767235 +0000 UTC m=+2147.744185566"
Oct 30 19:08:54.127210 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:54.126858    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 19:08:54.127210 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:54.126936    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs" podUID="dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 19:08:54.833296 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:54.831460    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:08:54.835261 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:54.835229    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 19:08:54.835405 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:54.835290    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl" podUID="2e3a7f70-606a-40b1-b541-e279a8438a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 19:08:55.128038 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:55.127591    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 19:08:55.128038 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:55.127655    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs" podUID="dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 19:08:55.841153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:55.836843    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 19:08:55.841153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:55.836905    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl" podUID="2e3a7f70-606a-40b1-b541-e279a8438a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 19:08:56.154208 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:56.153800    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 19:08:56.154208 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:56.153860    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs" podUID="dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 19:08:56.831711 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:56.831277    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 19:08:56.831711 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:56.831339    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl" podUID="2e3a7f70-606a-40b1-b541-e279a8438a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 19:08:57.134206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:57.133337    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 19:08:57.134206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:57.133426    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs" podUID="dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 19:08:57.836203 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:57.834615    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 19:08:57.836203 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:57.834677    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl" podUID="2e3a7f70-606a-40b1-b541-e279a8438a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 19:08:58.137202 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.133557    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 19:08:58.137202 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.133636    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs" podUID="dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 19:08:58.197243 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.197200    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"]
Oct 30 19:08:58.349437 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:08:58.349390    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:08:58.403056 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.402964    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:08:58.413654 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.413619    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689c23\"/\"kube-root-ca.crt\""
Oct 30 19:08:58.425443 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.425406    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-689c23\"/\"e2e-cmp-legacy-baf7a5-provider-cred\""
Oct 30 19:08:58.430297 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.426079    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689c23\"/\"routing-overlay\""
Oct 30 19:08:58.430297 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.426214    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689c23\"/\"openshift-service-ca.crt\""
Oct 30 19:08:58.430297 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.426279    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689c23\"/\"payload-processing-external-model-plugins-e2e-extmod-a-689c23\""
Oct 30 19:08:58.485925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.485878    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7971dc7a-3529-4b08-a901-87ad874c757f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-94565np2h\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:08:58.486276 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.486011    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7971dc7a-3529-4b08-a901-87ad874c757f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-94565np2h\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:08:58.486276 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.486174    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7971dc7a-3529-4b08-a901-87ad874c757f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-94565np2h\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:08:58.486444 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.486338    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbgbw\" (UniqueName: \"kubernetes.io/projected/7971dc7a-3529-4b08-a901-87ad874c757f-kube-api-access-vbgbw\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-94565np2h\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:08:58.510152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.510068    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:58.617784 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.590437    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:58.617784 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.592760    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7971dc7a-3529-4b08-a901-87ad874c757f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-94565np2h\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:08:58.617784 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.592884    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7971dc7a-3529-4b08-a901-87ad874c757f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-94565np2h\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:08:58.617784 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.593012    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7971dc7a-3529-4b08-a901-87ad874c757f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-94565np2h\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:08:58.617784 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.593169    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbgbw\" (UniqueName: \"kubernetes.io/projected/7971dc7a-3529-4b08-a901-87ad874c757f-kube-api-access-vbgbw\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-94565np2h\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:08:58.617784 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.606660    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7971dc7a-3529-4b08-a901-87ad874c757f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-94565np2h\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:08:58.635452 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.630799    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7971dc7a-3529-4b08-a901-87ad874c757f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-94565np2h\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:08:58.635452 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.633387    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7971dc7a-3529-4b08-a901-87ad874c757f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-94565np2h\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:08:58.704999 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.704903    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"]
Oct 30 19:08:58.742157 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.727966    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbgbw\" (UniqueName: \"kubernetes.io/projected/7971dc7a-3529-4b08-a901-87ad874c757f-kube-api-access-vbgbw\") pod \"payload-processing-external-model-e2e-extmod-a-689c23-94565np2h\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") " pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:08:58.832155 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.831488    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 19:08:58.832155 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.831582    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl" podUID="2e3a7f70-606a-40b1-b541-e279a8438a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 19:08:58.917149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.893464    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:58.917149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.903420    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:58.917149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.904761    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:58.925146 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.920204    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:58.962202 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:58.958164    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:08:59.213241 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.212024    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 19:08:59.213241 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.212117    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs" podUID="dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 19:08:59.266162 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.262312    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:59.623153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.606981    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:59.623153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.614210    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:08:59.623153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.617289    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 19:08:59.623153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.617371    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 19:08:59.935213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.934531    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:08:59.936489 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.935281    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 19:08:59.936489 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.936226    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 19:08:59.936489 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.936277    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl" podUID="2e3a7f70-606a-40b1-b541-e279a8438a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 19:08:59.938889 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.938863    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 19:08:59.944663 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.944632    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:59.945425 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.945376    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 19:08:59.973968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.973920    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 19:09:00.017152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:08:59.976238    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 19:09:00.017152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:00.006683    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 19:09:00.132154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:00.132085    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 19:09:00.135194 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:00.132883    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs" podUID="dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 19:09:00.274755 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:00.274726    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"]
Oct 30 19:09:00.651177 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:00.651084    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h" event={"ID":"7971dc7a-3529-4b08-a901-87ad874c757f","Type":"ContainerStarted","Data":"b7cbf22d4b923a157d07e740b90e6aa91c050c2f4d40161b2a0f886e0d0be38a"}
Oct 30 19:09:00.833151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:00.832764    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 19:09:00.833151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:00.832824    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl" podUID="2e3a7f70-606a-40b1-b541-e279a8438a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 19:09:00.833862 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:00.833838    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:09:01.133682 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:01.133570    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 19:09:01.133682 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:01.133640    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs" podUID="dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 19:09:01.663187 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:01.661593    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h" event={"ID":"7971dc7a-3529-4b08-a901-87ad874c757f","Type":"ContainerStarted","Data":"62577e2239ffd636ec5f88ed17f6d64da4e4731a715f93d2d278e271f8673450"}
Oct 30 19:09:01.663187 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:01.661689    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:09:01.829345 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:01.829314    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 19:09:01.829522 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:01.829378    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl" podUID="2e3a7f70-606a-40b1-b541-e279a8438a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 19:09:02.854315 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:02.853904    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:09:02.855863 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:02.855832    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:09:02.858253 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:02.858225    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:09:02.884939 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:02.884889    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h" podStartSLOduration=5.884876223 podStartE2EDuration="5.884876223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:01.710392306 +0000 UTC m=+2155.578810660" watchObservedRunningTime="2026-10-30 19:09:02.884876223 +0000 UTC m=+2156.753294553"
Oct 30 19:09:02.973785 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:02.973739    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:09:02.975566 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:02.975535    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:09:02.976485 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:02.976464    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:09:12.693626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:12.693583    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:09:19.500960 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:19.500925    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:09:22.390076 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:22.382227    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:27.774248 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:09:27.774182    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:09:37.769109 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:37.769074    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"]
Oct 30 19:09:37.769788 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:37.769276    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85" podUID="044b9f16-2b46-4246-b997-7791ba4fc3e8" containerName="istio-proxy" containerID="cri-o://30877e67666e50f0d78374b3757ddaca7a454ed45c39f659b3d3f3516e6a5c4c" gracePeriod=30
Oct 30 19:09:43.062848 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.062810    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:09:43.215621 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.215576    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-workload-socket\") pod \"044b9f16-2b46-4246-b997-7791ba4fc3e8\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") "
Oct 30 19:09:43.216298 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.216113    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-workload-socket" pod "044b9f16-2b46-4246-b997-7791ba4fc3e8" (UID: "044b9f16-2b46-4246-b997-7791ba4fc3e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:43.216529 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.216511    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-data\") pod \"044b9f16-2b46-4246-b997-7791ba4fc3e8\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") "
Oct 30 19:09:43.216662 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.216648    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-token\" (UniqueName: \"kubernetes.io/projected/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-token\") pod \"044b9f16-2b46-4246-b997-7791ba4fc3e8\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") "
Oct 30 19:09:43.216819 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.216806    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-envoy\") pod \"044b9f16-2b46-4246-b997-7791ba4fc3e8\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") "
Oct 30 19:09:43.218079 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.218045    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044b9f16-2b46-4246-b997-7791ba4fc3e8-kube-api-access-6tct5\" (UniqueName: \"kubernetes.io/projected/044b9f16-2b46-4246-b997-7791ba4fc3e8-kube-api-access-6tct5\") pod \"044b9f16-2b46-4246-b997-7791ba4fc3e8\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") "
Oct 30 19:09:43.232024 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.231983    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-data" pod "044b9f16-2b46-4246-b997-7791ba4fc3e8" (UID: "044b9f16-2b46-4246-b997-7791ba4fc3e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:43.232357 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.232291    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-podinfo\") pod \"044b9f16-2b46-4246-b997-7791ba4fc3e8\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") "
Oct 30 19:09:43.232717 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.232697    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-credential-socket\") pod \"044b9f16-2b46-4246-b997-7791ba4fc3e8\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") "
Oct 30 19:09:43.233230 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.233211    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/044b9f16-2b46-4246-b997-7791ba4fc3e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044b9f16-2b46-4246-b997-7791ba4fc3e8-istiod-ca-cert\") pod \"044b9f16-2b46-4246-b997-7791ba4fc3e8\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") "
Oct 30 19:09:43.233781 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.233096    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-credential-socket" pod "044b9f16-2b46-4246-b997-7791ba4fc3e8" (UID: "044b9f16-2b46-4246-b997-7791ba4fc3e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:43.236054 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.233816    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-workload-certs\") pod \"044b9f16-2b46-4246-b997-7791ba4fc3e8\" (UID: \"044b9f16-2b46-4246-b997-7791ba4fc3e8\") "
Oct 30 19:09:43.236054 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.233648    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044b9f16-2b46-4246-b997-7791ba4fc3e8-istiod-ca-cert" pod "044b9f16-2b46-4246-b997-7791ba4fc3e8" (UID: "044b9f16-2b46-4246-b997-7791ba4fc3e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:43.236054 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.234982    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044b9f16-2b46-4246-b997-7791ba4fc3e8-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:43.236054 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.235011    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:43.236054 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.235025    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:43.236054 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.235038    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:43.243859 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.243819    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-workload-certs" pod "044b9f16-2b46-4246-b997-7791ba4fc3e8" (UID: "044b9f16-2b46-4246-b997-7791ba4fc3e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:43.244740 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.244696    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044b9f16-2b46-4246-b997-7791ba4fc3e8-kube-api-access-6tct5" pod "044b9f16-2b46-4246-b997-7791ba4fc3e8" (UID: "044b9f16-2b46-4246-b997-7791ba4fc3e8"). InnerVolumeSpecName "kube-api-access-6tct5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:43.277158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.276382    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-podinfo" pod "044b9f16-2b46-4246-b997-7791ba4fc3e8" (UID: "044b9f16-2b46-4246-b997-7791ba4fc3e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:43.280991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.280233    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-token" pod "044b9f16-2b46-4246-b997-7791ba4fc3e8" (UID: "044b9f16-2b46-4246-b997-7791ba4fc3e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:43.280991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.280894    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-envoy" pod "044b9f16-2b46-4246-b997-7791ba4fc3e8" (UID: "044b9f16-2b46-4246-b997-7791ba4fc3e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:43.336879 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.336854    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:43.336879 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.336876    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:43.336879 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.336884    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:43.337138 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.336894    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tct5\" (UniqueName: \"kubernetes.io/projected/044b9f16-2b46-4246-b997-7791ba4fc3e8-kube-api-access-6tct5\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:43.337138 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.336904    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044b9f16-2b46-4246-b997-7791ba4fc3e8-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:43.630759 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.630717    4372 generic.go:356] "Generic (PLEG): container finished" podID="044b9f16-2b46-4246-b997-7791ba4fc3e8" containerID="30877e67666e50f0d78374b3757ddaca7a454ed45c39f659b3d3f3516e6a5c4c" exitCode=0
Oct 30 19:09:43.630983 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.630851    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85" event={"ID":"044b9f16-2b46-4246-b997-7791ba4fc3e8","Type":"ContainerDied","Data":"30877e67666e50f0d78374b3757ddaca7a454ed45c39f659b3d3f3516e6a5c4c"}
Oct 30 19:09:43.630983 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.630902    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85" event={"ID":"044b9f16-2b46-4246-b997-7791ba4fc3e8","Type":"ContainerDied","Data":"4778cea7c92018ff32b1fbd86ff1108f3864e00217f9fd054a947637e6931136"}
Oct 30 19:09:43.630983 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.630912    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"
Oct 30 19:09:43.630983 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.630927    4372 scope.go:122] "RemoveContainer" containerID="30877e67666e50f0d78374b3757ddaca7a454ed45c39f659b3d3f3516e6a5c4c"
Oct 30 19:09:43.647340 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.647317    4372 scope.go:122] "RemoveContainer" containerID="30877e67666e50f0d78374b3757ddaca7a454ed45c39f659b3d3f3516e6a5c4c"
Oct 30 19:09:43.647665 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:09:43.647645    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30877e67666e50f0d78374b3757ddaca7a454ed45c39f659b3d3f3516e6a5c4c\": container with ID starting with 30877e67666e50f0d78374b3757ddaca7a454ed45c39f659b3d3f3516e6a5c4c not found: ID does not exist" containerID="30877e67666e50f0d78374b3757ddaca7a454ed45c39f659b3d3f3516e6a5c4c"
Oct 30 19:09:43.647741 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.647672    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30877e67666e50f0d78374b3757ddaca7a454ed45c39f659b3d3f3516e6a5c4c"} err="failed to get container status \"30877e67666e50f0d78374b3757ddaca7a454ed45c39f659b3d3f3516e6a5c4c\": rpc error: code = NotFound desc = could not find container \"30877e67666e50f0d78374b3757ddaca7a454ed45c39f659b3d3f3516e6a5c4c\": container with ID starting with 30877e67666e50f0d78374b3757ddaca7a454ed45c39f659b3d3f3516e6a5c4c not found: ID does not exist"
Oct 30 19:09:43.715603 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.715579    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"]
Oct 30 19:09:43.720280 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:43.720255    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-fxm85"]
Oct 30 19:09:44.272468 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:09:44.272440    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="044b9f16-2b46-4246-b997-7791ba4fc3e8" path="/var/lib/kubelet/pods/044b9f16-2b46-4246-b997-7791ba4fc3e8/volumes"
Oct 30 19:09:57.698109 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:09:57.698028    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:10:16.026152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:16.022567    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:10:27.697396 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:10:27.697353    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:10:39.945641 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:39.945604    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"]
Oct 30 19:10:44.959439 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:44.959395    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h" podUID="7971dc7a-3529-4b08-a901-87ad874c757f" containerName="payload-processing" containerID="cri-o://62577e2239ffd636ec5f88ed17f6d64da4e4731a715f93d2d278e271f8673450" gracePeriod=25
Oct 30 19:10:45.266936 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.266908    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:10:45.377796 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.377765    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7971dc7a-3529-4b08-a901-87ad874c757f-kube-api-access-vbgbw\" (UniqueName: \"kubernetes.io/projected/7971dc7a-3529-4b08-a901-87ad874c757f-kube-api-access-vbgbw\") pod \"7971dc7a-3529-4b08-a901-87ad874c757f\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") "
Oct 30 19:10:45.378014 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.377995    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7971dc7a-3529-4b08-a901-87ad874c757f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7971dc7a-3529-4b08-a901-87ad874c757f-provider-credentials\") pod \"7971dc7a-3529-4b08-a901-87ad874c757f\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") "
Oct 30 19:10:45.378104 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.378089    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7971dc7a-3529-4b08-a901-87ad874c757f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7971dc7a-3529-4b08-a901-87ad874c757f-plugins-config-volume\") pod \"7971dc7a-3529-4b08-a901-87ad874c757f\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") "
Oct 30 19:10:45.378210 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.378195    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7971dc7a-3529-4b08-a901-87ad874c757f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7971dc7a-3529-4b08-a901-87ad874c757f-routing-overlay-volume\") pod \"7971dc7a-3529-4b08-a901-87ad874c757f\" (UID: \"7971dc7a-3529-4b08-a901-87ad874c757f\") "
Oct 30 19:10:45.378600 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.378515    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7971dc7a-3529-4b08-a901-87ad874c757f-plugins-config-volume" pod "7971dc7a-3529-4b08-a901-87ad874c757f" (UID: "7971dc7a-3529-4b08-a901-87ad874c757f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:45.378600 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.378588    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7971dc7a-3529-4b08-a901-87ad874c757f-routing-overlay-volume" pod "7971dc7a-3529-4b08-a901-87ad874c757f" (UID: "7971dc7a-3529-4b08-a901-87ad874c757f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:45.379356 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.379326    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7971dc7a-3529-4b08-a901-87ad874c757f-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:45.379486 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.379360    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7971dc7a-3529-4b08-a901-87ad874c757f-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:45.380377 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.380349    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7971dc7a-3529-4b08-a901-87ad874c757f-kube-api-access-vbgbw" pod "7971dc7a-3529-4b08-a901-87ad874c757f" (UID: "7971dc7a-3529-4b08-a901-87ad874c757f"). InnerVolumeSpecName "kube-api-access-vbgbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:45.380479 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.380451    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7971dc7a-3529-4b08-a901-87ad874c757f-provider-credentials" pod "7971dc7a-3529-4b08-a901-87ad874c757f" (UID: "7971dc7a-3529-4b08-a901-87ad874c757f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:45.480975 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.480896    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7971dc7a-3529-4b08-a901-87ad874c757f-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:45.480975 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.480925    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbgbw\" (UniqueName: \"kubernetes.io/projected/7971dc7a-3529-4b08-a901-87ad874c757f-kube-api-access-vbgbw\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:45.839976 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.839939    4372 generic.go:356] "Generic (PLEG): container finished" podID="7971dc7a-3529-4b08-a901-87ad874c757f" containerID="62577e2239ffd636ec5f88ed17f6d64da4e4731a715f93d2d278e271f8673450" exitCode=0
Oct 30 19:10:45.840171 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.839993    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"
Oct 30 19:10:45.840171 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.840019    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h" event={"ID":"7971dc7a-3529-4b08-a901-87ad874c757f","Type":"ContainerDied","Data":"62577e2239ffd636ec5f88ed17f6d64da4e4731a715f93d2d278e271f8673450"}
Oct 30 19:10:45.840171 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.840076    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h" event={"ID":"7971dc7a-3529-4b08-a901-87ad874c757f","Type":"ContainerDied","Data":"b7cbf22d4b923a157d07e740b90e6aa91c050c2f4d40161b2a0f886e0d0be38a"}
Oct 30 19:10:45.840171 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.840100    4372 scope.go:122] "RemoveContainer" containerID="62577e2239ffd636ec5f88ed17f6d64da4e4731a715f93d2d278e271f8673450"
Oct 30 19:10:45.863527 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.863503    4372 scope.go:122] "RemoveContainer" containerID="62577e2239ffd636ec5f88ed17f6d64da4e4731a715f93d2d278e271f8673450"
Oct 30 19:10:45.863813 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:10:45.863788    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62577e2239ffd636ec5f88ed17f6d64da4e4731a715f93d2d278e271f8673450\": container with ID starting with 62577e2239ffd636ec5f88ed17f6d64da4e4731a715f93d2d278e271f8673450 not found: ID does not exist" containerID="62577e2239ffd636ec5f88ed17f6d64da4e4731a715f93d2d278e271f8673450"
Oct 30 19:10:45.863862 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.863822    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62577e2239ffd636ec5f88ed17f6d64da4e4731a715f93d2d278e271f8673450"} err="failed to get container status \"62577e2239ffd636ec5f88ed17f6d64da4e4731a715f93d2d278e271f8673450\": rpc error: code = NotFound desc = could not find container \"62577e2239ffd636ec5f88ed17f6d64da4e4731a715f93d2d278e271f8673450\": container with ID starting with 62577e2239ffd636ec5f88ed17f6d64da4e4731a715f93d2d278e271f8673450 not found: ID does not exist"
Oct 30 19:10:45.890536 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.890502    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"]
Oct 30 19:10:45.902034 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:45.902010    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-689c23/payload-processing-external-model-e2e-extmod-a-689c23-94565np2h"]
Oct 30 19:10:46.297749 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:46.297667    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7971dc7a-3529-4b08-a901-87ad874c757f" path="/var/lib/kubelet/pods/7971dc7a-3529-4b08-a901-87ad874c757f/volumes"
Oct 30 19:10:49.799335 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:49.797837    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"]
Oct 30 19:10:54.798729 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:54.798690    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj" podUID="57458a83-1bee-4116-8b6e-5f511cc9d771" containerName="payload-processing" containerID="cri-o://fc350606c52f1294c823a4af9bdd8c1a154ace6a8e1d75ad84b46727b33d7ff1" gracePeriod=25
Oct 30 19:10:54.965957 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:54.965927    4372 generic.go:356] "Generic (PLEG): container finished" podID="57458a83-1bee-4116-8b6e-5f511cc9d771" containerID="fc350606c52f1294c823a4af9bdd8c1a154ace6a8e1d75ad84b46727b33d7ff1" exitCode=0
Oct 30 19:10:54.966098 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:54.966003    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj" event={"ID":"57458a83-1bee-4116-8b6e-5f511cc9d771","Type":"ContainerDied","Data":"fc350606c52f1294c823a4af9bdd8c1a154ace6a8e1d75ad84b46727b33d7ff1"}
Oct 30 19:10:55.148833 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.148801    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:10:55.226965 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.226939    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57458a83-1bee-4116-8b6e-5f511cc9d771-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57458a83-1bee-4116-8b6e-5f511cc9d771-plugins-config-volume\") pod \"57458a83-1bee-4116-8b6e-5f511cc9d771\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") "
Oct 30 19:10:55.227154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.227033    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57458a83-1bee-4116-8b6e-5f511cc9d771-provider-credentials\" (UniqueName: \"kubernetes.io/projected/57458a83-1bee-4116-8b6e-5f511cc9d771-provider-credentials\") pod \"57458a83-1bee-4116-8b6e-5f511cc9d771\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") "
Oct 30 19:10:55.227154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.227059    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57458a83-1bee-4116-8b6e-5f511cc9d771-kube-api-access-45dbq\" (UniqueName: \"kubernetes.io/projected/57458a83-1bee-4116-8b6e-5f511cc9d771-kube-api-access-45dbq\") pod \"57458a83-1bee-4116-8b6e-5f511cc9d771\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") "
Oct 30 19:10:55.227309 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.227183    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57458a83-1bee-4116-8b6e-5f511cc9d771-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/57458a83-1bee-4116-8b6e-5f511cc9d771-routing-overlay-volume\") pod \"57458a83-1bee-4116-8b6e-5f511cc9d771\" (UID: \"57458a83-1bee-4116-8b6e-5f511cc9d771\") "
Oct 30 19:10:55.227458 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.227433    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57458a83-1bee-4116-8b6e-5f511cc9d771-plugins-config-volume" pod "57458a83-1bee-4116-8b6e-5f511cc9d771" (UID: "57458a83-1bee-4116-8b6e-5f511cc9d771"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:55.227680 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.227647    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57458a83-1bee-4116-8b6e-5f511cc9d771-routing-overlay-volume" pod "57458a83-1bee-4116-8b6e-5f511cc9d771" (UID: "57458a83-1bee-4116-8b6e-5f511cc9d771"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:55.227861 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.227841    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/57458a83-1bee-4116-8b6e-5f511cc9d771-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:55.227922 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.227870    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57458a83-1bee-4116-8b6e-5f511cc9d771-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:55.229680 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.229657    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57458a83-1bee-4116-8b6e-5f511cc9d771-provider-credentials" pod "57458a83-1bee-4116-8b6e-5f511cc9d771" (UID: "57458a83-1bee-4116-8b6e-5f511cc9d771"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:55.229796 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.229694    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57458a83-1bee-4116-8b6e-5f511cc9d771-kube-api-access-45dbq" pod "57458a83-1bee-4116-8b6e-5f511cc9d771" (UID: "57458a83-1bee-4116-8b6e-5f511cc9d771"). InnerVolumeSpecName "kube-api-access-45dbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:55.357154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.354142    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/57458a83-1bee-4116-8b6e-5f511cc9d771-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:55.357154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.354177    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45dbq\" (UniqueName: \"kubernetes.io/projected/57458a83-1bee-4116-8b6e-5f511cc9d771-kube-api-access-45dbq\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:55.978743 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.978713    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"
Oct 30 19:10:55.979314 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.978745    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj" event={"ID":"57458a83-1bee-4116-8b6e-5f511cc9d771","Type":"ContainerDied","Data":"efeef7dce6f96f9d36c5f7a031a0bb8c7bf99b34afde7d44cd0bf99402ba1b8e"}
Oct 30 19:10:55.979314 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:55.978826    4372 scope.go:122] "RemoveContainer" containerID="fc350606c52f1294c823a4af9bdd8c1a154ace6a8e1d75ad84b46727b33d7ff1"
Oct 30 19:10:56.010258 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:56.010226    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"]
Oct 30 19:10:56.017233 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:56.017211    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-cdbbpm9tj"]
Oct 30 19:10:56.267583 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:10:56.267519    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57458a83-1bee-4116-8b6e-5f511cc9d771" path="/var/lib/kubelet/pods/57458a83-1bee-4116-8b6e-5f511cc9d771/volumes"
Oct 30 19:10:57.696714 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:10:57.696676    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:11:03.996904 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:03.996870    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"]
Oct 30 19:11:03.997695 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:03.997680    4372 cpu_manager.go:414] "containerMap: removing container" podUID="044b9f16-2b46-4246-b997-7791ba4fc3e8" containerName="istio-proxy"
Oct 30 19:11:03.997748 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:03.997698    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044b9f16-2b46-4246-b997-7791ba4fc3e8" containerName="istio-proxy"
Oct 30 19:11:03.997748 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:03.997732    4372 cpu_manager.go:414] "containerMap: removing container" podUID="57458a83-1bee-4116-8b6e-5f511cc9d771" containerName="payload-processing"
Oct 30 19:11:03.997748 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:03.997737    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57458a83-1bee-4116-8b6e-5f511cc9d771" containerName="payload-processing"
Oct 30 19:11:03.997849 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:03.997757    4372 cpu_manager.go:414] "containerMap: removing container" podUID="7971dc7a-3529-4b08-a901-87ad874c757f" containerName="payload-processing"
Oct 30 19:11:03.997849 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:03.997763    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7971dc7a-3529-4b08-a901-87ad874c757f" containerName="payload-processing"
Oct 30 19:11:03.997914 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:03.997867    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="7971dc7a-3529-4b08-a901-87ad874c757f" containerName="payload-processing"
Oct 30 19:11:03.997914 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:03.997877    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="044b9f16-2b46-4246-b997-7791ba4fc3e8" containerName="istio-proxy"
Oct 30 19:11:03.997914 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:03.997890    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="57458a83-1bee-4116-8b6e-5f511cc9d771" containerName="payload-processing"
Oct 30 19:11:04.001798 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.001777    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:04.010639 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.010597    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d72beb\""
Oct 30 19:11:04.010783 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.010679    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"openshift-service-ca.crt\""
Oct 30 19:11:04.011825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.011803    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"routing-overlay\""
Oct 30 19:11:04.012036 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.012014    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"kube-root-ca.crt\""
Oct 30 19:11:04.012248 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.012236    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"agc-cred-54fc64a67d3d\""
Oct 30 19:11:04.012424 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.012405    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"agc-cred-bda27d1a8117\""
Oct 30 19:11:04.015365 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.015345    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"agc-cred-94b1b3df02aa\""
Oct 30 19:11:04.015365 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.015346    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"payload-processing-external-model-e2e--b99c0f48-dockercfg-hq7ks\""
Oct 30 19:11:04.019179 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.019159    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"]
Oct 30 19:11:04.048042 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.048018    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9e19ee-c302-49d2-9fb1-2220fc68127b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:04.048259 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.048239    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e9e19ee-c302-49d2-9fb1-2220fc68127b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:04.048323 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.048305    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e9e19ee-c302-49d2-9fb1-2220fc68127b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:04.048418 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.048398    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gpk2\" (UniqueName: \"kubernetes.io/projected/2e9e19ee-c302-49d2-9fb1-2220fc68127b-kube-api-access-7gpk2\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:04.149806 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.149774    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e9e19ee-c302-49d2-9fb1-2220fc68127b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:04.149974 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.149839    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e9e19ee-c302-49d2-9fb1-2220fc68127b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:04.149974 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.149868    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gpk2\" (UniqueName: \"kubernetes.io/projected/2e9e19ee-c302-49d2-9fb1-2220fc68127b-kube-api-access-7gpk2\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:04.150090 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.150032    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9e19ee-c302-49d2-9fb1-2220fc68127b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:04.150587 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.150562    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e9e19ee-c302-49d2-9fb1-2220fc68127b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:04.150733 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.150711    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9e19ee-c302-49d2-9fb1-2220fc68127b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:04.152682 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.152662    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e9e19ee-c302-49d2-9fb1-2220fc68127b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:04.168449 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.168430    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gpk2\" (UniqueName: \"kubernetes.io/projected/2e9e19ee-c302-49d2-9fb1-2220fc68127b-kube-api-access-7gpk2\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:04.319581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.319511    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:04.628553 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:11:04.628522    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9e19ee_c302_49d2_9fb1_2220fc68127b.slice/crio-2b989a4e5c6e9b7bf62bfd97364aca353e94b6bc720a33666470ba2548dd1471 WatchSource:0}: Error finding container 2b989a4e5c6e9b7bf62bfd97364aca353e94b6bc720a33666470ba2548dd1471: Status 404 returned error can't find the container with id 2b989a4e5c6e9b7bf62bfd97364aca353e94b6bc720a33666470ba2548dd1471
Oct 30 19:11:04.629249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:04.629218    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"]
Oct 30 19:11:05.100442 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:05.100406    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj" event={"ID":"2e9e19ee-c302-49d2-9fb1-2220fc68127b","Type":"ContainerStarted","Data":"8b280b565c3faa629810fb181fbf061439d32810136bf17a8a492bd9e3a5707e"}
Oct 30 19:11:05.100910 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:05.100470    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:05.100910 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:05.100493    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj" event={"ID":"2e9e19ee-c302-49d2-9fb1-2220fc68127b","Type":"ContainerStarted","Data":"2b989a4e5c6e9b7bf62bfd97364aca353e94b6bc720a33666470ba2548dd1471"}
Oct 30 19:11:05.124705 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:05.124658    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj" podStartSLOduration=2.124643972 podStartE2EDuration="2.124643972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:05.120291909 +0000 UTC m=+2278.988710244" watchObservedRunningTime="2026-10-30 19:11:05.124643972 +0000 UTC m=+2278.993062303"
Oct 30 19:11:10.201152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.200296    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"]
Oct 30 19:11:10.249198 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.248682    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"]
Oct 30 19:11:10.249198 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.248792    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:10.254568 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.254546    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"kube-root-ca.crt\""
Oct 30 19:11:10.254783 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.254759    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"openshift-service-ca.crt\""
Oct 30 19:11:10.254783 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.254777    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"routing-overlay\""
Oct 30 19:11:10.256270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.256248    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4e94aa\""
Oct 30 19:11:10.256401 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.256291    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"agc-cred-394aab6f23b5\""
Oct 30 19:11:10.256401 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.256326    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"payload-processing-external-model-e2e--0d1ddd5a-dockercfg-7twb9\""
Oct 30 19:11:10.406648 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.406607    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7z8m\" (UniqueName: \"kubernetes.io/projected/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-kube-api-access-v7z8m\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:10.406816 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.406754    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:10.406898 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.406830    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:10.406960 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.406946    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:10.523205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.513643    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7z8m\" (UniqueName: \"kubernetes.io/projected/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-kube-api-access-v7z8m\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:10.523205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.513832    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:10.523205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.513926    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:10.523205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.514004    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:10.523205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.515988    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:10.523205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.516580    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:10.559149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.558933    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7z8m\" (UniqueName: \"kubernetes.io/projected/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-kube-api-access-v7z8m\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:10.581149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.574987    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:10.865406 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:10.865372    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:11.253861 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:11:11.253829    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd410ca1f_24e3_4b4f_aa9e_1af0010cccf5.slice/crio-b33d7b2e6489baef528a52b693ae1b12e2b8355b93438178892c0b97e518ebe2 WatchSource:0}: Error finding container b33d7b2e6489baef528a52b693ae1b12e2b8355b93438178892c0b97e518ebe2: Status 404 returned error can't find the container with id b33d7b2e6489baef528a52b693ae1b12e2b8355b93438178892c0b97e518ebe2
Oct 30 19:11:11.255537 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:11.255477    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"]
Oct 30 19:11:12.270163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:12.270109    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5" event={"ID":"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5","Type":"ContainerStarted","Data":"0beb8ee697afddb276d49552d1bfda518c2c8ae338ee1ec96fd314f502c206ad"}
Oct 30 19:11:12.270637 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:12.270189    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:12.270637 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:12.270217    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5" event={"ID":"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5","Type":"ContainerStarted","Data":"b33d7b2e6489baef528a52b693ae1b12e2b8355b93438178892c0b97e518ebe2"}
Oct 30 19:11:12.296078 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:12.296019    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5" podStartSLOduration=2.296002589 podStartE2EDuration="2.296002589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:12.295617719 +0000 UTC m=+2286.164036071" watchObservedRunningTime="2026-10-30 19:11:12.296002589 +0000 UTC m=+2286.164420920"
Oct 30 19:11:13.987986 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:13.987952    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:11:16.154152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:16.152410    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:11:16.961860 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:16.955264    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:16.971335 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:16.971265    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:11:16.973501 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:16.973458    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 19:11:16.979383 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:16.979330    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:11:16.984595 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:16.984564    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:21.857413 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:21.857371    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"]
Oct 30 19:11:21.861064 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:21.861036    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:26.858003 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:26.857963    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5" podUID="d410ca1f-24e3-4b4f-aa9e-1af0010cccf5" containerName="payload-processing" containerID="cri-o://0beb8ee697afddb276d49552d1bfda518c2c8ae338ee1ec96fd314f502c206ad" gracePeriod=25
Oct 30 19:11:27.227440 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.227403    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:27.250219 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.250181    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-plugins-config-volume\") pod \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") "
Oct 30 19:11:27.250468 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.250448    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-provider-credentials\") pod \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") "
Oct 30 19:11:27.250568 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.250542    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-kube-api-access-v7z8m\" (UniqueName: \"kubernetes.io/projected/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-kube-api-access-v7z8m\") pod \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") "
Oct 30 19:11:27.250721 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.250705    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-routing-overlay-volume\") pod \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\" (UID: \"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5\") "
Oct 30 19:11:27.250930 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.250900    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-plugins-config-volume" pod "d410ca1f-24e3-4b4f-aa9e-1af0010cccf5" (UID: "d410ca1f-24e3-4b4f-aa9e-1af0010cccf5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:27.251532 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.251494    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-routing-overlay-volume" pod "d410ca1f-24e3-4b4f-aa9e-1af0010cccf5" (UID: "d410ca1f-24e3-4b4f-aa9e-1af0010cccf5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:27.251870 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.251850    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:27.252045 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.252029    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:27.254325 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.253892    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-kube-api-access-v7z8m" pod "d410ca1f-24e3-4b4f-aa9e-1af0010cccf5" (UID: "d410ca1f-24e3-4b4f-aa9e-1af0010cccf5"). InnerVolumeSpecName "kube-api-access-v7z8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:27.255683 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.254777    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-provider-credentials" pod "d410ca1f-24e3-4b4f-aa9e-1af0010cccf5" (UID: "d410ca1f-24e3-4b4f-aa9e-1af0010cccf5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:27.354169 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.353952    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:27.354169 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.353983    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7z8m\" (UniqueName: \"kubernetes.io/projected/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5-kube-api-access-v7z8m\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:27.698096 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:11:27.698060    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:11:27.871499 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.871467    4372 generic.go:356] "Generic (PLEG): container finished" podID="d410ca1f-24e3-4b4f-aa9e-1af0010cccf5" containerID="0beb8ee697afddb276d49552d1bfda518c2c8ae338ee1ec96fd314f502c206ad" exitCode=0
Oct 30 19:11:27.871985 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.871522    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"
Oct 30 19:11:27.871985 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.871560    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5" event={"ID":"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5","Type":"ContainerDied","Data":"0beb8ee697afddb276d49552d1bfda518c2c8ae338ee1ec96fd314f502c206ad"}
Oct 30 19:11:27.871985 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.871630    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5" event={"ID":"d410ca1f-24e3-4b4f-aa9e-1af0010cccf5","Type":"ContainerDied","Data":"b33d7b2e6489baef528a52b693ae1b12e2b8355b93438178892c0b97e518ebe2"}
Oct 30 19:11:27.871985 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.871669    4372 scope.go:122] "RemoveContainer" containerID="0beb8ee697afddb276d49552d1bfda518c2c8ae338ee1ec96fd314f502c206ad"
Oct 30 19:11:27.883276 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.883258    4372 scope.go:122] "RemoveContainer" containerID="0beb8ee697afddb276d49552d1bfda518c2c8ae338ee1ec96fd314f502c206ad"
Oct 30 19:11:27.883520 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:11:27.883503    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0beb8ee697afddb276d49552d1bfda518c2c8ae338ee1ec96fd314f502c206ad\": container with ID starting with 0beb8ee697afddb276d49552d1bfda518c2c8ae338ee1ec96fd314f502c206ad not found: ID does not exist" containerID="0beb8ee697afddb276d49552d1bfda518c2c8ae338ee1ec96fd314f502c206ad"
Oct 30 19:11:27.883574 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.883528    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0beb8ee697afddb276d49552d1bfda518c2c8ae338ee1ec96fd314f502c206ad"} err="failed to get container status \"0beb8ee697afddb276d49552d1bfda518c2c8ae338ee1ec96fd314f502c206ad\": rpc error: code = NotFound desc = could not find container \"0beb8ee697afddb276d49552d1bfda518c2c8ae338ee1ec96fd314f502c206ad\": container with ID starting with 0beb8ee697afddb276d49552d1bfda518c2c8ae338ee1ec96fd314f502c206ad not found: ID does not exist"
Oct 30 19:11:27.901574 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.901555    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"]
Oct 30 19:11:27.905932 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:27.905914    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-7854qcsh5"]
Oct 30 19:11:28.267555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:28.267525    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d410ca1f-24e3-4b4f-aa9e-1af0010cccf5" path="/var/lib/kubelet/pods/d410ca1f-24e3-4b4f-aa9e-1af0010cccf5/volumes"
Oct 30 19:11:41.436143 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.436102    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"]
Oct 30 19:11:41.436958 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.436941    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d410ca1f-24e3-4b4f-aa9e-1af0010cccf5" containerName="payload-processing"
Oct 30 19:11:41.437010 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.436959    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d410ca1f-24e3-4b4f-aa9e-1af0010cccf5" containerName="payload-processing"
Oct 30 19:11:41.437085 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.437076    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d410ca1f-24e3-4b4f-aa9e-1af0010cccf5" containerName="payload-processing"
Oct 30 19:11:41.441296 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.441278    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:41.451922 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.451300    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"e2e-cfl-740f4f-provider-cred\""
Oct 30 19:11:41.451922 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.451410    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"payload-processing-external-model-e2e--307b9dfd-dockercfg-q6kdn\""
Oct 30 19:11:41.452977 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.452933    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e8c554\""
Oct 30 19:11:41.497281 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.488804    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"routing-overlay\""
Oct 30 19:11:41.519604 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.491379    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"kube-root-ca.crt\""
Oct 30 19:11:41.519819 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.491467    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8c554\"/\"openshift-service-ca.crt\""
Oct 30 19:11:41.563150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.552428    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:41.563150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.552511    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zshlq\" (UniqueName: \"kubernetes.io/projected/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-kube-api-access-zshlq\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:41.563150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.552588    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:41.563150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.552922    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:41.578358 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.578323    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"]
Oct 30 19:11:41.699266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.698691    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:41.699266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.699070    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:41.699266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.699205    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:41.699573 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.699266    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshlq\" (UniqueName: \"kubernetes.io/projected/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-kube-api-access-zshlq\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:41.779221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.770231    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:41.804089 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.804036    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zshlq\" (UniqueName: \"kubernetes.io/projected/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-kube-api-access-zshlq\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:41.804876 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.804847    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:41.835871 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:41.835831    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") " pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:42.119613 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:42.119582    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:42.707163 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:11:42.704456    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31e1c9c_cfc4_49e1_bcc0_2c490fa5878e.slice/crio-1d952754d1d3f70af88c19d04e37c1644aacf9e15c05e1f2116421ca95e76afd WatchSource:0}: Error finding container 1d952754d1d3f70af88c19d04e37c1644aacf9e15c05e1f2116421ca95e76afd: Status 404 returned error can't find the container with id 1d952754d1d3f70af88c19d04e37c1644aacf9e15c05e1f2116421ca95e76afd
Oct 30 19:11:42.771153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:42.754119    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"]
Oct 30 19:11:43.327994 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:43.327960    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p" event={"ID":"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e","Type":"ContainerStarted","Data":"683e1c4530e8ac3231b8f5d3c7f5d86da157d7b559ef4daf6cb4a16f3ff92662"}
Oct 30 19:11:43.328300 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:43.328280    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p" event={"ID":"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e","Type":"ContainerStarted","Data":"1d952754d1d3f70af88c19d04e37c1644aacf9e15c05e1f2116421ca95e76afd"}
Oct 30 19:11:43.329448 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:43.329402    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:43.355179 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:43.354624    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p" podStartSLOduration=2.354606982 podStartE2EDuration="2.354606982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:43.349409187 +0000 UTC m=+2317.217827520" watchObservedRunningTime="2026-10-30 19:11:43.354606982 +0000 UTC m=+2317.223025314"
Oct 30 19:11:47.805148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:47.804886    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"]
Oct 30 19:11:47.847287 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:47.842391    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:11:47.987150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:47.977246    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0441de8d-4f95-4a47-9145-f0c537025c12-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:11:47.999242 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:47.994324    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0441de8d-4f95-4a47-9145-f0c537025c12-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:11:47.999242 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:47.994475    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0441de8d-4f95-4a47-9145-f0c537025c12-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:11:47.999242 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:47.994525    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4hvn\" (UniqueName: \"kubernetes.io/projected/0441de8d-4f95-4a47-9145-f0c537025c12-kube-api-access-t4hvn\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:11:48.097641 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:48.096917    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"]
Oct 30 19:11:48.101015 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:48.100977    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0441de8d-4f95-4a47-9145-f0c537025c12-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:11:48.101270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:48.101052    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4hvn\" (UniqueName: \"kubernetes.io/projected/0441de8d-4f95-4a47-9145-f0c537025c12-kube-api-access-t4hvn\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:11:48.101355 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:48.101298    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0441de8d-4f95-4a47-9145-f0c537025c12-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:11:48.101590 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:48.101547    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0441de8d-4f95-4a47-9145-f0c537025c12-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:11:48.102864 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:48.102834    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0441de8d-4f95-4a47-9145-f0c537025c12-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:11:48.103240 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:48.103204    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0441de8d-4f95-4a47-9145-f0c537025c12-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:11:48.109207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:48.109180    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0441de8d-4f95-4a47-9145-f0c537025c12-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:11:48.119976 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:48.119511    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4hvn\" (UniqueName: \"kubernetes.io/projected/0441de8d-4f95-4a47-9145-f0c537025c12-kube-api-access-t4hvn\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:11:48.373206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:48.369390    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:11:48.826546 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:48.826516    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"]
Oct 30 19:11:48.826904 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:11:48.826568    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0441de8d_4f95_4a47_9145_f0c537025c12.slice/crio-1390c7f6fdc3114db80a183c5b0aeb46a428aa8ff1b42c97bf31a769c9ca48ef WatchSource:0}: Error finding container 1390c7f6fdc3114db80a183c5b0aeb46a428aa8ff1b42c97bf31a769c9ca48ef: Status 404 returned error can't find the container with id 1390c7f6fdc3114db80a183c5b0aeb46a428aa8ff1b42c97bf31a769c9ca48ef
Oct 30 19:11:49.572930 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:49.572852    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb" event={"ID":"0441de8d-4f95-4a47-9145-f0c537025c12","Type":"ContainerStarted","Data":"cda8e7d56b5b81bfdabdc2c4d4e926c6987c373ef60b966c6420c8af0607eea7"}
Oct 30 19:11:49.575079 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:49.573041    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:11:49.575079 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:49.573096    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb" event={"ID":"0441de8d-4f95-4a47-9145-f0c537025c12","Type":"ContainerStarted","Data":"1390c7f6fdc3114db80a183c5b0aeb46a428aa8ff1b42c97bf31a769c9ca48ef"}
Oct 30 19:11:49.597513 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:49.597470    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb" podStartSLOduration=2.597457664 podStartE2EDuration="2.597457664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:49.593244649 +0000 UTC m=+2323.461662981" watchObservedRunningTime="2026-10-30 19:11:49.597457664 +0000 UTC m=+2323.465875994"
Oct 30 19:11:55.346208 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:55.346177    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:11:57.760319 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:11:57.760275    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:11:59.527792 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:59.527754    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"]
Oct 30 19:11:59.535160 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:59.535069    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"]
Oct 30 19:11:59.544513 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:11:59.544489    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:12:04.528624 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:04.528588    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj" podUID="2e9e19ee-c302-49d2-9fb1-2220fc68127b" containerName="payload-processing" containerID="cri-o://8b280b565c3faa629810fb181fbf061439d32810136bf17a8a492bd9e3a5707e" gracePeriod=25
Oct 30 19:12:04.536152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:04.536098    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb" podUID="0441de8d-4f95-4a47-9145-f0c537025c12" containerName="payload-processing" containerID="cri-o://cda8e7d56b5b81bfdabdc2c4d4e926c6987c373ef60b966c6420c8af0607eea7" gracePeriod=25
Oct 30 19:12:04.780977 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:04.780890    4372 generic.go:356] "Generic (PLEG): container finished" podID="2e9e19ee-c302-49d2-9fb1-2220fc68127b" containerID="8b280b565c3faa629810fb181fbf061439d32810136bf17a8a492bd9e3a5707e" exitCode=0
Oct 30 19:12:04.781168 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:04.781056    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj" event={"ID":"2e9e19ee-c302-49d2-9fb1-2220fc68127b","Type":"ContainerDied","Data":"8b280b565c3faa629810fb181fbf061439d32810136bf17a8a492bd9e3a5707e"}
Oct 30 19:12:04.783411 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:04.783376    4372 generic.go:356] "Generic (PLEG): container finished" podID="0441de8d-4f95-4a47-9145-f0c537025c12" containerID="cda8e7d56b5b81bfdabdc2c4d4e926c6987c373ef60b966c6420c8af0607eea7" exitCode=0
Oct 30 19:12:04.783533 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:04.783504    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb" event={"ID":"0441de8d-4f95-4a47-9145-f0c537025c12","Type":"ContainerDied","Data":"cda8e7d56b5b81bfdabdc2c4d4e926c6987c373ef60b966c6420c8af0607eea7"}
Oct 30 19:12:04.877137 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:04.877093    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:12:04.904159 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:04.902386    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:12:05.021794 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.021758    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0441de8d-4f95-4a47-9145-f0c537025c12-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0441de8d-4f95-4a47-9145-f0c537025c12-provider-credentials\") pod \"0441de8d-4f95-4a47-9145-f0c537025c12\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") "
Oct 30 19:12:05.021988 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.021825    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e9e19ee-c302-49d2-9fb1-2220fc68127b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9e19ee-c302-49d2-9fb1-2220fc68127b-plugins-config-volume\") pod \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") "
Oct 30 19:12:05.021988 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.021853    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e9e19ee-c302-49d2-9fb1-2220fc68127b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e9e19ee-c302-49d2-9fb1-2220fc68127b-provider-credentials\") pod \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") "
Oct 30 19:12:05.021988 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.021962    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e9e19ee-c302-49d2-9fb1-2220fc68127b-kube-api-access-7gpk2\" (UniqueName: \"kubernetes.io/projected/2e9e19ee-c302-49d2-9fb1-2220fc68127b-kube-api-access-7gpk2\") pod \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") "
Oct 30 19:12:05.022300 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.022036    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e9e19ee-c302-49d2-9fb1-2220fc68127b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e9e19ee-c302-49d2-9fb1-2220fc68127b-routing-overlay-volume\") pod \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\" (UID: \"2e9e19ee-c302-49d2-9fb1-2220fc68127b\") "
Oct 30 19:12:05.022300 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.022241    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0441de8d-4f95-4a47-9145-f0c537025c12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0441de8d-4f95-4a47-9145-f0c537025c12-plugins-config-volume\") pod \"0441de8d-4f95-4a47-9145-f0c537025c12\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") "
Oct 30 19:12:05.022426 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.022329    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0441de8d-4f95-4a47-9145-f0c537025c12-kube-api-access-t4hvn\" (UniqueName: \"kubernetes.io/projected/0441de8d-4f95-4a47-9145-f0c537025c12-kube-api-access-t4hvn\") pod \"0441de8d-4f95-4a47-9145-f0c537025c12\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") "
Oct 30 19:12:05.022426 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.022403    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0441de8d-4f95-4a47-9145-f0c537025c12-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0441de8d-4f95-4a47-9145-f0c537025c12-routing-overlay-volume\") pod \"0441de8d-4f95-4a47-9145-f0c537025c12\" (UID: \"0441de8d-4f95-4a47-9145-f0c537025c12\") "
Oct 30 19:12:05.022957 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.022924    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9e19ee-c302-49d2-9fb1-2220fc68127b-routing-overlay-volume" pod "2e9e19ee-c302-49d2-9fb1-2220fc68127b" (UID: "2e9e19ee-c302-49d2-9fb1-2220fc68127b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:05.023368 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.023297    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0441de8d-4f95-4a47-9145-f0c537025c12-plugins-config-volume" pod "0441de8d-4f95-4a47-9145-f0c537025c12" (UID: "0441de8d-4f95-4a47-9145-f0c537025c12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:05.023471 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.023301    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0441de8d-4f95-4a47-9145-f0c537025c12-routing-overlay-volume" pod "0441de8d-4f95-4a47-9145-f0c537025c12" (UID: "0441de8d-4f95-4a47-9145-f0c537025c12"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:05.023557 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.023516    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9e19ee-c302-49d2-9fb1-2220fc68127b-plugins-config-volume" pod "2e9e19ee-c302-49d2-9fb1-2220fc68127b" (UID: "2e9e19ee-c302-49d2-9fb1-2220fc68127b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:05.023630 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.023553    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e9e19ee-c302-49d2-9fb1-2220fc68127b-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:05.025079 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.025050    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0441de8d-4f95-4a47-9145-f0c537025c12-provider-credentials" pod "0441de8d-4f95-4a47-9145-f0c537025c12" (UID: "0441de8d-4f95-4a47-9145-f0c537025c12"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:05.025254 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.025188    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e9e19ee-c302-49d2-9fb1-2220fc68127b-provider-credentials" pod "2e9e19ee-c302-49d2-9fb1-2220fc68127b" (UID: "2e9e19ee-c302-49d2-9fb1-2220fc68127b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:05.025726 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.025703    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e9e19ee-c302-49d2-9fb1-2220fc68127b-kube-api-access-7gpk2" pod "2e9e19ee-c302-49d2-9fb1-2220fc68127b" (UID: "2e9e19ee-c302-49d2-9fb1-2220fc68127b"). InnerVolumeSpecName "kube-api-access-7gpk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:05.026392 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.026366    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0441de8d-4f95-4a47-9145-f0c537025c12-kube-api-access-t4hvn" pod "0441de8d-4f95-4a47-9145-f0c537025c12" (UID: "0441de8d-4f95-4a47-9145-f0c537025c12"). InnerVolumeSpecName "kube-api-access-t4hvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:05.124797 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.124771    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0441de8d-4f95-4a47-9145-f0c537025c12-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:05.124797 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.124797    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4hvn\" (UniqueName: \"kubernetes.io/projected/0441de8d-4f95-4a47-9145-f0c537025c12-kube-api-access-t4hvn\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:05.124955 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.124807    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0441de8d-4f95-4a47-9145-f0c537025c12-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:05.124955 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.124816    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0441de8d-4f95-4a47-9145-f0c537025c12-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:05.124955 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.124828    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9e19ee-c302-49d2-9fb1-2220fc68127b-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:05.124955 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.124841    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e9e19ee-c302-49d2-9fb1-2220fc68127b-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:05.124955 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.124856    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gpk2\" (UniqueName: \"kubernetes.io/projected/2e9e19ee-c302-49d2-9fb1-2220fc68127b-kube-api-access-7gpk2\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:05.792636 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.792601    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"
Oct 30 19:12:05.792636 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.792628    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb" event={"ID":"0441de8d-4f95-4a47-9145-f0c537025c12","Type":"ContainerDied","Data":"1390c7f6fdc3114db80a183c5b0aeb46a428aa8ff1b42c97bf31a769c9ca48ef"}
Oct 30 19:12:05.793187 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.792690    4372 scope.go:122] "RemoveContainer" containerID="cda8e7d56b5b81bfdabdc2c4d4e926c6987c373ef60b966c6420c8af0607eea7"
Oct 30 19:12:05.794520 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.794498    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj" event={"ID":"2e9e19ee-c302-49d2-9fb1-2220fc68127b","Type":"ContainerDied","Data":"2b989a4e5c6e9b7bf62bfd97364aca353e94b6bc720a33666470ba2548dd1471"}
Oct 30 19:12:05.794625 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.794511    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"
Oct 30 19:12:05.805383 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.805364    4372 scope.go:122] "RemoveContainer" containerID="8b280b565c3faa629810fb181fbf061439d32810136bf17a8a492bd9e3a5707e"
Oct 30 19:12:05.826098 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.826069    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"]
Oct 30 19:12:05.831972 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.831949    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-79cfkfxzb"]
Oct 30 19:12:05.846184 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.846161    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"]
Oct 30 19:12:05.850643 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:05.850612    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-76b52xrgj"]
Oct 30 19:12:06.268556 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:06.268525    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0441de8d-4f95-4a47-9145-f0c537025c12" path="/var/lib/kubelet/pods/0441de8d-4f95-4a47-9145-f0c537025c12/volumes"
Oct 30 19:12:06.268964 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:06.268949    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e9e19ee-c302-49d2-9fb1-2220fc68127b" path="/var/lib/kubelet/pods/2e9e19ee-c302-49d2-9fb1-2220fc68127b/volumes"
Oct 30 19:12:13.405843 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:13.405806    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:12:15.791576 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:15.791529    4372 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:15.875483 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:15.875433    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"]
Oct 30 19:12:15.884782 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:15.884723    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh" podUID="616634c4-e125-41b1-82e3-db2556beadb9" containerName="istio-proxy" containerID="cri-o://5674d90315d1186c624820cc6f9b005bd541025e92883a49611b9422ea03877b" gracePeriod=30
Oct 30 19:12:15.950445 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:15.949300    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:15.950445 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:15.950057    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:16.101463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:16.097252    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:12:16.101463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:16.098489    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:16.958984 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:16.958938    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:16.962348 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:16.962321    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 19:12:17.078620 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:17.078338    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 19:12:22.181005 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.180945    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:12:22.292853 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.292814    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-credential-socket\") pod \"616634c4-e125-41b1-82e3-db2556beadb9\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") "
Oct 30 19:12:22.293024 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.292906    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-istio-envoy\") pod \"616634c4-e125-41b1-82e3-db2556beadb9\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") "
Oct 30 19:12:22.293089 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.293022    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616634c4-e125-41b1-82e3-db2556beadb9-istio-token\" (UniqueName: \"kubernetes.io/projected/616634c4-e125-41b1-82e3-db2556beadb9-istio-token\") pod \"616634c4-e125-41b1-82e3-db2556beadb9\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") "
Oct 30 19:12:22.293089 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.293069    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-istio-data\") pod \"616634c4-e125-41b1-82e3-db2556beadb9\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") "
Oct 30 19:12:22.293232 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.293100    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/616634c4-e125-41b1-82e3-db2556beadb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616634c4-e125-41b1-82e3-db2556beadb9-istio-podinfo\") pod \"616634c4-e125-41b1-82e3-db2556beadb9\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") "
Oct 30 19:12:22.293232 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.293116    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-credential-socket" pod "616634c4-e125-41b1-82e3-db2556beadb9" (UID: "616634c4-e125-41b1-82e3-db2556beadb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:22.293356 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.293242    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616634c4-e125-41b1-82e3-db2556beadb9-kube-api-access-fr489\" (UniqueName: \"kubernetes.io/projected/616634c4-e125-41b1-82e3-db2556beadb9-kube-api-access-fr489\") pod \"616634c4-e125-41b1-82e3-db2556beadb9\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") "
Oct 30 19:12:22.293356 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.293298    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-workload-socket\") pod \"616634c4-e125-41b1-82e3-db2556beadb9\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") "
Oct 30 19:12:22.293462 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.293380    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-workload-certs\") pod \"616634c4-e125-41b1-82e3-db2556beadb9\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") "
Oct 30 19:12:22.294232 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.293543    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/616634c4-e125-41b1-82e3-db2556beadb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616634c4-e125-41b1-82e3-db2556beadb9-istiod-ca-cert\") pod \"616634c4-e125-41b1-82e3-db2556beadb9\" (UID: \"616634c4-e125-41b1-82e3-db2556beadb9\") "
Oct 30 19:12:22.294232 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.293538    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-istio-data" pod "616634c4-e125-41b1-82e3-db2556beadb9" (UID: "616634c4-e125-41b1-82e3-db2556beadb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:22.294232 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.293851    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-workload-certs" pod "616634c4-e125-41b1-82e3-db2556beadb9" (UID: "616634c4-e125-41b1-82e3-db2556beadb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:22.294232 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.294204    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616634c4-e125-41b1-82e3-db2556beadb9-istiod-ca-cert" pod "616634c4-e125-41b1-82e3-db2556beadb9" (UID: "616634c4-e125-41b1-82e3-db2556beadb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:22.294232 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.294220    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-workload-socket" pod "616634c4-e125-41b1-82e3-db2556beadb9" (UID: "616634c4-e125-41b1-82e3-db2556beadb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:22.294626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.294560    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.294626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.294582    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.294626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.294598    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.294626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.294612    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.294859 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.294630    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616634c4-e125-41b1-82e3-db2556beadb9-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.295765 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.295744    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-istio-envoy" pod "616634c4-e125-41b1-82e3-db2556beadb9" (UID: "616634c4-e125-41b1-82e3-db2556beadb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:22.295958 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.295935    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/616634c4-e125-41b1-82e3-db2556beadb9-istio-podinfo" pod "616634c4-e125-41b1-82e3-db2556beadb9" (UID: "616634c4-e125-41b1-82e3-db2556beadb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:12:22.296254 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.296228    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616634c4-e125-41b1-82e3-db2556beadb9-istio-token" pod "616634c4-e125-41b1-82e3-db2556beadb9" (UID: "616634c4-e125-41b1-82e3-db2556beadb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:22.296412 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.296391    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616634c4-e125-41b1-82e3-db2556beadb9-kube-api-access-fr489" pod "616634c4-e125-41b1-82e3-db2556beadb9" (UID: "616634c4-e125-41b1-82e3-db2556beadb9"). InnerVolumeSpecName "kube-api-access-fr489". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:22.399151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.397248    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616634c4-e125-41b1-82e3-db2556beadb9-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.399151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.397288    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616634c4-e125-41b1-82e3-db2556beadb9-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.399151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.397307    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616634c4-e125-41b1-82e3-db2556beadb9-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.399151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.397321    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr489\" (UniqueName: \"kubernetes.io/projected/616634c4-e125-41b1-82e3-db2556beadb9-kube-api-access-fr489\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.730520 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.730482    4372 generic.go:356] "Generic (PLEG): container finished" podID="616634c4-e125-41b1-82e3-db2556beadb9" containerID="5674d90315d1186c624820cc6f9b005bd541025e92883a49611b9422ea03877b" exitCode=0
Oct 30 19:12:22.730726 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.730533    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"
Oct 30 19:12:22.730726 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.730567    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh" event={"ID":"616634c4-e125-41b1-82e3-db2556beadb9","Type":"ContainerDied","Data":"5674d90315d1186c624820cc6f9b005bd541025e92883a49611b9422ea03877b"}
Oct 30 19:12:22.730726 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.730640    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh" event={"ID":"616634c4-e125-41b1-82e3-db2556beadb9","Type":"ContainerDied","Data":"d2569f001e24f40f9900750f13a4f2f6b71083ef1896ef7ff57a4a20c0ff56c2"}
Oct 30 19:12:22.730726 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.730672    4372 scope.go:122] "RemoveContainer" containerID="5674d90315d1186c624820cc6f9b005bd541025e92883a49611b9422ea03877b"
Oct 30 19:12:22.745749 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.745725    4372 scope.go:122] "RemoveContainer" containerID="5674d90315d1186c624820cc6f9b005bd541025e92883a49611b9422ea03877b"
Oct 30 19:12:22.746058 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:12:22.746037    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5674d90315d1186c624820cc6f9b005bd541025e92883a49611b9422ea03877b\": container with ID starting with 5674d90315d1186c624820cc6f9b005bd541025e92883a49611b9422ea03877b not found: ID does not exist" containerID="5674d90315d1186c624820cc6f9b005bd541025e92883a49611b9422ea03877b"
Oct 30 19:12:22.746158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.746067    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5674d90315d1186c624820cc6f9b005bd541025e92883a49611b9422ea03877b"} err="failed to get container status \"5674d90315d1186c624820cc6f9b005bd541025e92883a49611b9422ea03877b\": rpc error: code = NotFound desc = could not find container \"5674d90315d1186c624820cc6f9b005bd541025e92883a49611b9422ea03877b\": container with ID starting with 5674d90315d1186c624820cc6f9b005bd541025e92883a49611b9422ea03877b not found: ID does not exist"
Oct 30 19:12:22.762027 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.762003    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"]
Oct 30 19:12:22.767172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:22.767150    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-8bfvh"]
Oct 30 19:12:24.267945 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:12:24.267908    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="616634c4-e125-41b1-82e3-db2556beadb9" path="/var/lib/kubelet/pods/616634c4-e125-41b1-82e3-db2556beadb9/volumes"
Oct 30 19:12:27.697827 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:12:27.697793    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:12:57.697762 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:12:57.697673    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:13:09.867076 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:09.867041    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:13:23.990920 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:23.990885    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"]
Oct 30 19:13:27.697386 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:13:27.697348    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:13:28.995118 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:28.995059    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p" podUID="d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e" containerName="payload-processing" containerID="cri-o://683e1c4530e8ac3231b8f5d3c7f5d86da157d7b559ef4daf6cb4a16f3ff92662" gracePeriod=25
Oct 30 19:13:29.517250 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.517224    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:13:29.619976 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.619949    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-routing-overlay-volume\") pod \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") "
Oct 30 19:13:29.620150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.620008    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-kube-api-access-zshlq\" (UniqueName: \"kubernetes.io/projected/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-kube-api-access-zshlq\") pod \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") "
Oct 30 19:13:29.620150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.620102    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-provider-credentials\") pod \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") "
Oct 30 19:13:29.620247 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.620147    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-plugins-config-volume\") pod \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\" (UID: \"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e\") "
Oct 30 19:13:29.620412 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.620384    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-routing-overlay-volume" pod "d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e" (UID: "d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:29.620536 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.620509    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-plugins-config-volume" pod "d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e" (UID: "d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:29.620905 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.620886    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:29.620961 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.620910    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:29.622514 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.622491    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-provider-credentials" pod "d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e" (UID: "d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:29.622605 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.622589    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-kube-api-access-zshlq" pod "d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e" (UID: "d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e"). InnerVolumeSpecName "kube-api-access-zshlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:29.722679 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.722609    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zshlq\" (UniqueName: \"kubernetes.io/projected/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-kube-api-access-zshlq\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:29.722679 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.722645    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:29.893283 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.893246    4372 generic.go:356] "Generic (PLEG): container finished" podID="d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e" containerID="683e1c4530e8ac3231b8f5d3c7f5d86da157d7b559ef4daf6cb4a16f3ff92662" exitCode=0
Oct 30 19:13:29.893455 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.893312    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"
Oct 30 19:13:29.893455 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.893332    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p" event={"ID":"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e","Type":"ContainerDied","Data":"683e1c4530e8ac3231b8f5d3c7f5d86da157d7b559ef4daf6cb4a16f3ff92662"}
Oct 30 19:13:29.893455 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.893409    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p" event={"ID":"d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e","Type":"ContainerDied","Data":"1d952754d1d3f70af88c19d04e37c1644aacf9e15c05e1f2116421ca95e76afd"}
Oct 30 19:13:29.893455 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.893432    4372 scope.go:122] "RemoveContainer" containerID="683e1c4530e8ac3231b8f5d3c7f5d86da157d7b559ef4daf6cb4a16f3ff92662"
Oct 30 19:13:29.905170 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.905151    4372 scope.go:122] "RemoveContainer" containerID="683e1c4530e8ac3231b8f5d3c7f5d86da157d7b559ef4daf6cb4a16f3ff92662"
Oct 30 19:13:29.905472 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:13:29.905448    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683e1c4530e8ac3231b8f5d3c7f5d86da157d7b559ef4daf6cb4a16f3ff92662\": container with ID starting with 683e1c4530e8ac3231b8f5d3c7f5d86da157d7b559ef4daf6cb4a16f3ff92662 not found: ID does not exist" containerID="683e1c4530e8ac3231b8f5d3c7f5d86da157d7b559ef4daf6cb4a16f3ff92662"
Oct 30 19:13:29.905564 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.905479    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683e1c4530e8ac3231b8f5d3c7f5d86da157d7b559ef4daf6cb4a16f3ff92662"} err="failed to get container status \"683e1c4530e8ac3231b8f5d3c7f5d86da157d7b559ef4daf6cb4a16f3ff92662\": rpc error: code = NotFound desc = could not find container \"683e1c4530e8ac3231b8f5d3c7f5d86da157d7b559ef4daf6cb4a16f3ff92662\": container with ID starting with 683e1c4530e8ac3231b8f5d3c7f5d86da157d7b559ef4daf6cb4a16f3ff92662 not found: ID does not exist"
Oct 30 19:13:29.927720 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.927694    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"]
Oct 30 19:13:29.932050 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:29.932025    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e8c554/payload-processing-external-model-e2e-extmod-a-e8c554-5fd54g62p"]
Oct 30 19:13:30.336313 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:30.333526    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e" path="/var/lib/kubelet/pods/d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e/volumes"
Oct 30 19:13:50.634163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:50.633577    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"]
Oct 30 19:13:50.634163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:50.633834    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy" containerID="cri-o://4d8a3e297e336a81c8c4866e17921957b58235f6d5a346893c41673517b79106" gracePeriod=30
Oct 30 19:13:55.150741 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:55.150708    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"]
Oct 30 19:13:55.151270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:55.150931    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg" podUID="a588c60a-b15e-4d10-a2cc-71eb915006db" containerName="istio-proxy" containerID="cri-o://6a3302c9cbd0f68b7b35a214174641da9162469b7d91f92539ac8d1a8ebf1db4" gracePeriod=30
Oct 30 19:13:55.945880 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:55.945854    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 19:13:56.046616 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.046533    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10111dcc-0436-4206-b798-d0bac119328a-istio-token\" (UniqueName: \"kubernetes.io/projected/10111dcc-0436-4206-b798-d0bac119328a-istio-token\") pod \"10111dcc-0436-4206-b798-d0bac119328a\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") "
Oct 30 19:13:56.046616 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.046599    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-credential-socket\") pod \"10111dcc-0436-4206-b798-d0bac119328a\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") "
Oct 30 19:13:56.046871 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.046709    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10111dcc-0436-4206-b798-d0bac119328a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10111dcc-0436-4206-b798-d0bac119328a-istiod-ca-cert\") pod \"10111dcc-0436-4206-b798-d0bac119328a\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") "
Oct 30 19:13:56.046871 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.046780    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10111dcc-0436-4206-b798-d0bac119328a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10111dcc-0436-4206-b798-d0bac119328a-istio-podinfo\") pod \"10111dcc-0436-4206-b798-d0bac119328a\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") "
Oct 30 19:13:56.046871 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.046860    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-credential-socket" pod "10111dcc-0436-4206-b798-d0bac119328a" (UID: "10111dcc-0436-4206-b798-d0bac119328a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:56.047045 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.046866    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-istio-envoy\") pod \"10111dcc-0436-4206-b798-d0bac119328a\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") "
Oct 30 19:13:56.047045 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.046927    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-workload-socket\") pod \"10111dcc-0436-4206-b798-d0bac119328a\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") "
Oct 30 19:13:56.047257 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.047222    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10111dcc-0436-4206-b798-d0bac119328a-kube-api-access-hxzxj\" (UniqueName: \"kubernetes.io/projected/10111dcc-0436-4206-b798-d0bac119328a-kube-api-access-hxzxj\") pod \"10111dcc-0436-4206-b798-d0bac119328a\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") "
Oct 30 19:13:56.047367 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.047253    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10111dcc-0436-4206-b798-d0bac119328a-istiod-ca-cert" pod "10111dcc-0436-4206-b798-d0bac119328a" (UID: "10111dcc-0436-4206-b798-d0bac119328a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:56.047367 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.047262    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-workload-certs\") pod \"10111dcc-0436-4206-b798-d0bac119328a\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") "
Oct 30 19:13:56.047478 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.047444    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-istio-data\") pod \"10111dcc-0436-4206-b798-d0bac119328a\" (UID: \"10111dcc-0436-4206-b798-d0bac119328a\") "
Oct 30 19:13:56.048519 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.048294    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-istio-data" pod "10111dcc-0436-4206-b798-d0bac119328a" (UID: "10111dcc-0436-4206-b798-d0bac119328a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:56.048519 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.048437    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.048519 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.048453    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.048519 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.048467    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10111dcc-0436-4206-b798-d0bac119328a-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.050939 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.050913    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-workload-certs" pod "10111dcc-0436-4206-b798-d0bac119328a" (UID: "10111dcc-0436-4206-b798-d0bac119328a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:56.051494 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.051443    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-workload-socket" pod "10111dcc-0436-4206-b798-d0bac119328a" (UID: "10111dcc-0436-4206-b798-d0bac119328a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:56.051606 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.051504    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10111dcc-0436-4206-b798-d0bac119328a-istio-podinfo" pod "10111dcc-0436-4206-b798-d0bac119328a" (UID: "10111dcc-0436-4206-b798-d0bac119328a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:56.051751 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.051722    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10111dcc-0436-4206-b798-d0bac119328a-kube-api-access-hxzxj" pod "10111dcc-0436-4206-b798-d0bac119328a" (UID: "10111dcc-0436-4206-b798-d0bac119328a"). InnerVolumeSpecName "kube-api-access-hxzxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:56.051751 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.051733    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-istio-envoy" pod "10111dcc-0436-4206-b798-d0bac119328a" (UID: "10111dcc-0436-4206-b798-d0bac119328a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:56.051953 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.051917    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10111dcc-0436-4206-b798-d0bac119328a-istio-token" pod "10111dcc-0436-4206-b798-d0bac119328a" (UID: "10111dcc-0436-4206-b798-d0bac119328a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:56.150037 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.150003    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10111dcc-0436-4206-b798-d0bac119328a-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.150037 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.150036    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.150278 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.150045    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.150278 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.150055    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxzxj\" (UniqueName: \"kubernetes.io/projected/10111dcc-0436-4206-b798-d0bac119328a-kube-api-access-hxzxj\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.150278 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.150065    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10111dcc-0436-4206-b798-d0bac119328a-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.150278 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.150074    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10111dcc-0436-4206-b798-d0bac119328a-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.716756 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.716720    4372 generic.go:356] "Generic (PLEG): container finished" podID="10111dcc-0436-4206-b798-d0bac119328a" containerID="4d8a3e297e336a81c8c4866e17921957b58235f6d5a346893c41673517b79106" exitCode=0
Oct 30 19:13:56.717267 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.716777    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"
Oct 30 19:13:56.717267 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.716811    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" event={"ID":"10111dcc-0436-4206-b798-d0bac119328a","Type":"ContainerDied","Data":"4d8a3e297e336a81c8c4866e17921957b58235f6d5a346893c41673517b79106"}
Oct 30 19:13:56.717267 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.716875    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb" event={"ID":"10111dcc-0436-4206-b798-d0bac119328a","Type":"ContainerDied","Data":"2b7804859edc34d9dda5af3c37312e1783735d9e34b16e3ecdddd86606621e31"}
Oct 30 19:13:56.717267 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.716898    4372 scope.go:122] "RemoveContainer" containerID="4d8a3e297e336a81c8c4866e17921957b58235f6d5a346893c41673517b79106"
Oct 30 19:13:56.731057 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.731032    4372 scope.go:122] "RemoveContainer" containerID="704cc894c079ca154a3adda2c659990411c5e2ad0a197eccc1026637ff020c3e"
Oct 30 19:13:56.743139 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.743108    4372 scope.go:122] "RemoveContainer" containerID="4d8a3e297e336a81c8c4866e17921957b58235f6d5a346893c41673517b79106"
Oct 30 19:13:56.743492 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:13:56.743464    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d8a3e297e336a81c8c4866e17921957b58235f6d5a346893c41673517b79106\": container with ID starting with 4d8a3e297e336a81c8c4866e17921957b58235f6d5a346893c41673517b79106 not found: ID does not exist" containerID="4d8a3e297e336a81c8c4866e17921957b58235f6d5a346893c41673517b79106"
Oct 30 19:13:56.743569 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.743503    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d8a3e297e336a81c8c4866e17921957b58235f6d5a346893c41673517b79106"} err="failed to get container status \"4d8a3e297e336a81c8c4866e17921957b58235f6d5a346893c41673517b79106\": rpc error: code = NotFound desc = could not find container \"4d8a3e297e336a81c8c4866e17921957b58235f6d5a346893c41673517b79106\": container with ID starting with 4d8a3e297e336a81c8c4866e17921957b58235f6d5a346893c41673517b79106 not found: ID does not exist"
Oct 30 19:13:56.743569 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.743527    4372 scope.go:122] "RemoveContainer" containerID="704cc894c079ca154a3adda2c659990411c5e2ad0a197eccc1026637ff020c3e"
Oct 30 19:13:56.743806 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:13:56.743776    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704cc894c079ca154a3adda2c659990411c5e2ad0a197eccc1026637ff020c3e\": container with ID starting with 704cc894c079ca154a3adda2c659990411c5e2ad0a197eccc1026637ff020c3e not found: ID does not exist" containerID="704cc894c079ca154a3adda2c659990411c5e2ad0a197eccc1026637ff020c3e"
Oct 30 19:13:56.743869 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.743816    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704cc894c079ca154a3adda2c659990411c5e2ad0a197eccc1026637ff020c3e"} err="failed to get container status \"704cc894c079ca154a3adda2c659990411c5e2ad0a197eccc1026637ff020c3e\": rpc error: code = NotFound desc = could not find container \"704cc894c079ca154a3adda2c659990411c5e2ad0a197eccc1026637ff020c3e\": container with ID starting with 704cc894c079ca154a3adda2c659990411c5e2ad0a197eccc1026637ff020c3e not found: ID does not exist"
Oct 30 19:13:56.744531 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.744511    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"]
Oct 30 19:13:56.750173 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:56.750153    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-bh9zb"]
Oct 30 19:13:57.697702 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:13:57.697662    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:13:58.295153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:13:58.294436    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10111dcc-0436-4206-b798-d0bac119328a" path="/var/lib/kubelet/pods/10111dcc-0436-4206-b798-d0bac119328a/volumes"
Oct 30 19:14:00.453170 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.453110    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 19:14:00.549555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.549521    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-data\") pod \"a588c60a-b15e-4d10-a2cc-71eb915006db\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") "
Oct 30 19:14:00.549721 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.549624    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a588c60a-b15e-4d10-a2cc-71eb915006db-kube-api-access-vhc5g\" (UniqueName: \"kubernetes.io/projected/a588c60a-b15e-4d10-a2cc-71eb915006db-kube-api-access-vhc5g\") pod \"a588c60a-b15e-4d10-a2cc-71eb915006db\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") "
Oct 30 19:14:00.549721 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.549694    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-credential-socket\") pod \"a588c60a-b15e-4d10-a2cc-71eb915006db\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") "
Oct 30 19:14:00.549832 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.549798    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-data" pod "a588c60a-b15e-4d10-a2cc-71eb915006db" (UID: "a588c60a-b15e-4d10-a2cc-71eb915006db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:00.549896 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.549856    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-podinfo\") pod \"a588c60a-b15e-4d10-a2cc-71eb915006db\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") "
Oct 30 19:14:00.549896 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.549889    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a588c60a-b15e-4d10-a2cc-71eb915006db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a588c60a-b15e-4d10-a2cc-71eb915006db-istiod-ca-cert\") pod \"a588c60a-b15e-4d10-a2cc-71eb915006db\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") "
Oct 30 19:14:00.550014 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.549910    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-token\" (UniqueName: \"kubernetes.io/projected/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-token\") pod \"a588c60a-b15e-4d10-a2cc-71eb915006db\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") "
Oct 30 19:14:00.550014 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.549990    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-workload-certs\") pod \"a588c60a-b15e-4d10-a2cc-71eb915006db\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") "
Oct 30 19:14:00.550144 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.550019    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-workload-socket\") pod \"a588c60a-b15e-4d10-a2cc-71eb915006db\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") "
Oct 30 19:14:00.550144 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.550056    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-credential-socket" pod "a588c60a-b15e-4d10-a2cc-71eb915006db" (UID: "a588c60a-b15e-4d10-a2cc-71eb915006db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:00.550144 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.550079    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-envoy\") pod \"a588c60a-b15e-4d10-a2cc-71eb915006db\" (UID: \"a588c60a-b15e-4d10-a2cc-71eb915006db\") "
Oct 30 19:14:00.550319 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.550268    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a588c60a-b15e-4d10-a2cc-71eb915006db-istiod-ca-cert" pod "a588c60a-b15e-4d10-a2cc-71eb915006db" (UID: "a588c60a-b15e-4d10-a2cc-71eb915006db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:00.550767 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.550704    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-workload-socket" pod "a588c60a-b15e-4d10-a2cc-71eb915006db" (UID: "a588c60a-b15e-4d10-a2cc-71eb915006db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:00.551306 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.551146    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:00.551306 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.551179    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a588c60a-b15e-4d10-a2cc-71eb915006db-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:00.551306 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.551194    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:00.551306 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.551209    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:00.551306 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.551279    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-workload-certs" pod "a588c60a-b15e-4d10-a2cc-71eb915006db" (UID: "a588c60a-b15e-4d10-a2cc-71eb915006db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:00.552568 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.552545    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-envoy" pod "a588c60a-b15e-4d10-a2cc-71eb915006db" (UID: "a588c60a-b15e-4d10-a2cc-71eb915006db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:00.552880 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.552861    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-podinfo" pod "a588c60a-b15e-4d10-a2cc-71eb915006db" (UID: "a588c60a-b15e-4d10-a2cc-71eb915006db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:00.552959 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.552918    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a588c60a-b15e-4d10-a2cc-71eb915006db-kube-api-access-vhc5g" pod "a588c60a-b15e-4d10-a2cc-71eb915006db" (UID: "a588c60a-b15e-4d10-a2cc-71eb915006db"). InnerVolumeSpecName "kube-api-access-vhc5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:00.554416 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.554391    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-token" pod "a588c60a-b15e-4d10-a2cc-71eb915006db" (UID: "a588c60a-b15e-4d10-a2cc-71eb915006db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:00.652558 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.652524    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:00.652558 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.652556    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:00.652558 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.652567    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:00.652810 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.652576    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a588c60a-b15e-4d10-a2cc-71eb915006db-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:00.652810 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.652585    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhc5g\" (UniqueName: \"kubernetes.io/projected/a588c60a-b15e-4d10-a2cc-71eb915006db-kube-api-access-vhc5g\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:00.752689 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.752593    4372 generic.go:356] "Generic (PLEG): container finished" podID="a588c60a-b15e-4d10-a2cc-71eb915006db" containerID="6a3302c9cbd0f68b7b35a214174641da9162469b7d91f92539ac8d1a8ebf1db4" exitCode=0
Oct 30 19:14:00.752689 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.752637    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg" event={"ID":"a588c60a-b15e-4d10-a2cc-71eb915006db","Type":"ContainerDied","Data":"6a3302c9cbd0f68b7b35a214174641da9162469b7d91f92539ac8d1a8ebf1db4"}
Oct 30 19:14:00.752981 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.752705    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg" event={"ID":"a588c60a-b15e-4d10-a2cc-71eb915006db","Type":"ContainerDied","Data":"9cbc573dba2805e62b104a74078c14cf1fcfa74c64638c7229c7c64c38eabe3f"}
Oct 30 19:14:00.752981 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.752738    4372 scope.go:122] "RemoveContainer" containerID="6a3302c9cbd0f68b7b35a214174641da9162469b7d91f92539ac8d1a8ebf1db4"
Oct 30 19:14:00.752981 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.752647    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"
Oct 30 19:14:00.839825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.839695    4372 scope.go:122] "RemoveContainer" containerID="6a3302c9cbd0f68b7b35a214174641da9162469b7d91f92539ac8d1a8ebf1db4"
Oct 30 19:14:00.840191 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:14:00.840103    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a3302c9cbd0f68b7b35a214174641da9162469b7d91f92539ac8d1a8ebf1db4\": container with ID starting with 6a3302c9cbd0f68b7b35a214174641da9162469b7d91f92539ac8d1a8ebf1db4 not found: ID does not exist" containerID="6a3302c9cbd0f68b7b35a214174641da9162469b7d91f92539ac8d1a8ebf1db4"
Oct 30 19:14:00.840317 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.840265    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3302c9cbd0f68b7b35a214174641da9162469b7d91f92539ac8d1a8ebf1db4"} err="failed to get container status \"6a3302c9cbd0f68b7b35a214174641da9162469b7d91f92539ac8d1a8ebf1db4\": rpc error: code = NotFound desc = could not find container \"6a3302c9cbd0f68b7b35a214174641da9162469b7d91f92539ac8d1a8ebf1db4\": container with ID starting with 6a3302c9cbd0f68b7b35a214174641da9162469b7d91f92539ac8d1a8ebf1db4 not found: ID does not exist"
Oct 30 19:14:00.866171 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.866095    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"]
Oct 30 19:14:00.870880 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:00.870857    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-fl7cg"]
Oct 30 19:14:02.268447 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:02.268409    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a588c60a-b15e-4d10-a2cc-71eb915006db" path="/var/lib/kubelet/pods/a588c60a-b15e-4d10-a2cc-71eb915006db/volumes"
Oct 30 19:14:10.313982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.313937    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"]
Oct 30 19:14:10.315875 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.315841    4372 cpu_manager.go:414] "containerMap: removing container" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy"
Oct 30 19:14:10.315875 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.315875    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy"
Oct 30 19:14:10.316081 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.315914    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e" containerName="payload-processing"
Oct 30 19:14:10.316081 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.315922    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e" containerName="payload-processing"
Oct 30 19:14:10.316081 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.315965    4372 cpu_manager.go:414] "containerMap: removing container" podUID="616634c4-e125-41b1-82e3-db2556beadb9" containerName="istio-proxy"
Oct 30 19:14:10.316081 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.315974    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616634c4-e125-41b1-82e3-db2556beadb9" containerName="istio-proxy"
Oct 30 19:14:10.316081 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.316009    4372 cpu_manager.go:414] "containerMap: removing container" podUID="0441de8d-4f95-4a47-9145-f0c537025c12" containerName="payload-processing"
Oct 30 19:14:10.316081 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.316016    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0441de8d-4f95-4a47-9145-f0c537025c12" containerName="payload-processing"
Oct 30 19:14:10.316081 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.316047    4372 cpu_manager.go:414] "containerMap: removing container" podUID="2e9e19ee-c302-49d2-9fb1-2220fc68127b" containerName="payload-processing"
Oct 30 19:14:10.316081 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.316055    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e9e19ee-c302-49d2-9fb1-2220fc68127b" containerName="payload-processing"
Oct 30 19:14:10.316548 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.316153    4372 cpu_manager.go:414] "containerMap: removing container" podUID="a588c60a-b15e-4d10-a2cc-71eb915006db" containerName="istio-proxy"
Oct 30 19:14:10.316548 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.316164    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a588c60a-b15e-4d10-a2cc-71eb915006db" containerName="istio-proxy"
Oct 30 19:14:10.316652 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.316557    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="a588c60a-b15e-4d10-a2cc-71eb915006db" containerName="istio-proxy"
Oct 30 19:14:10.316652 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.316597    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy"
Oct 30 19:14:10.316652 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.316611    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31e1c9c-cfc4-49e1-bcc0-2c490fa5878e" containerName="payload-processing"
Oct 30 19:14:10.316652 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.316639    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="616634c4-e125-41b1-82e3-db2556beadb9" containerName="istio-proxy"
Oct 30 19:14:10.316859 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.316651    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="0441de8d-4f95-4a47-9145-f0c537025c12" containerName="payload-processing"
Oct 30 19:14:10.316859 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.316663    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e9e19ee-c302-49d2-9fb1-2220fc68127b" containerName="payload-processing"
Oct 30 19:14:10.316859 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.316682    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy"
Oct 30 19:14:10.339782 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.339751    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.346558 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.346529    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"]
Oct 30 19:14:10.438231 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.438200    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.438401 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.438252    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd868454-02ed-4227-ae09-be713e3473d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.438401 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.438325    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.438501 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.438421    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.438501 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.438438    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.438501 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.438468    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.438501 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.438493    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd868454-02ed-4227-ae09-be713e3473d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.438684 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.438660    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd868454-02ed-4227-ae09-be713e3473d1-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.438732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.438699    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmmfj\" (UniqueName: \"kubernetes.io/projected/dd868454-02ed-4227-ae09-be713e3473d1-kube-api-access-qmmfj\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.539696 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.539667    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.539696 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.539696    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.539913 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.539724    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.539913 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.539754    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd868454-02ed-4227-ae09-be713e3473d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.539913 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.539854    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd868454-02ed-4227-ae09-be713e3473d1-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.539913 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.539888    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmmfj\" (UniqueName: \"kubernetes.io/projected/dd868454-02ed-4227-ae09-be713e3473d1-kube-api-access-qmmfj\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.540118 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.540004    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.540118 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.540059    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd868454-02ed-4227-ae09-be713e3473d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.540118 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.540099    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.540325 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.540107    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.540325 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.540257    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.540447 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.540430    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.540562 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.540540    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.540808 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.540791    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd868454-02ed-4227-ae09-be713e3473d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.542324 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.542303    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.542554 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.542534    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd868454-02ed-4227-ae09-be713e3473d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.558787 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.558761    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd868454-02ed-4227-ae09-be713e3473d1-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.558886 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.558864    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmmfj\" (UniqueName: \"kubernetes.io/projected/dd868454-02ed-4227-ae09-be713e3473d1-kube-api-access-qmmfj\") pod \"maas-default-gateway-openshift-default-855f9574cf-bjk2q\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.658398 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.658367    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:10.954079 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:14:10.954050    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd868454_02ed_4227_ae09_be713e3473d1.slice/crio-c914b9b1e221febbfc89f0db0cb9ccd05d3c5602cba51a67d9b226b9970565bc WatchSource:0}: Error finding container c914b9b1e221febbfc89f0db0cb9ccd05d3c5602cba51a67d9b226b9970565bc: Status 404 returned error can't find the container with id c914b9b1e221febbfc89f0db0cb9ccd05d3c5602cba51a67d9b226b9970565bc
Oct 30 19:14:10.954673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.954650    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"]
Oct 30 19:14:10.956969 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.956933    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:14:10.957096 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.957026    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:14:10.957096 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:10.957076    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:14:11.972476 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:11.972437    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q" event={"ID":"dd868454-02ed-4227-ae09-be713e3473d1","Type":"ContainerStarted","Data":"8f6b27eb0dfa4b8eae0c6d9ae94467a7bff03fd3249ab78071aed8d3ab6efc86"}
Oct 30 19:14:11.973054 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:11.973028    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q" event={"ID":"dd868454-02ed-4227-ae09-be713e3473d1","Type":"ContainerStarted","Data":"c914b9b1e221febbfc89f0db0cb9ccd05d3c5602cba51a67d9b226b9970565bc"}
Oct 30 19:14:12.005709 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:12.005650    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q" podStartSLOduration=2.005632185 podStartE2EDuration="2.005632185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:11.999443479 +0000 UTC m=+2465.867861811" watchObservedRunningTime="2026-10-30 19:14:12.005632185 +0000 UTC m=+2465.874050517"
Oct 30 19:14:12.658787 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:12.658755    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:12.660079 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:12.660055    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-bjk2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 19:14:12.660226 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:12.660112    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q" podUID="dd868454-02ed-4227-ae09-be713e3473d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 19:14:13.658722 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:13.658684    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-bjk2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 19:14:13.659305 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:13.658739    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q" podUID="dd868454-02ed-4227-ae09-be713e3473d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 19:14:14.661657 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:14.661606    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-bjk2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 19:14:14.662314 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:14.662260    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q" podUID="dd868454-02ed-4227-ae09-be713e3473d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 19:14:15.659760 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:15.659724    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-bjk2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 19:14:15.659954 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:15.659780    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q" podUID="dd868454-02ed-4227-ae09-be713e3473d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 19:14:16.659765 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:16.658993    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-bjk2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 19:14:16.659765 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:16.659067    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q" podUID="dd868454-02ed-4227-ae09-be713e3473d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 19:14:17.658740 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:17.658699    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-bjk2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 19:14:17.658925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:17.658763    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q" podUID="dd868454-02ed-4227-ae09-be713e3473d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 19:14:18.659423 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:18.659378    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-bjk2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 19:14:18.659996 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:18.659434    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q" podUID="dd868454-02ed-4227-ae09-be713e3473d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 19:14:19.659660 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:19.659620    4372 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-bjk2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 19:14:19.660063 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:19.659696    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q" podUID="dd868454-02ed-4227-ae09-be713e3473d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 19:14:21.281451 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:21.281376    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:21.282828 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:21.282799    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:21.283708 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:21.283690    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:14:22.419007 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:22.418975    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"]
Oct 30 19:14:22.419504 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:22.419202    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46" podUID="4290d2e4-ccae-445f-b184-4509020ddefa" containerName="istio-proxy" containerID="cri-o://5ebb28d2fe5028ff8ce1765c3c2d47f37101fc1b630a62e3001b25edea512f11" gracePeriod=30
Oct 30 19:14:23.739538 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:23.739507    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:14:28.039152 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:14:28.037652    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:14:28.611142 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.611076    4372 generic.go:356] "Generic (PLEG): container finished" podID="4290d2e4-ccae-445f-b184-4509020ddefa" containerID="5ebb28d2fe5028ff8ce1765c3c2d47f37101fc1b630a62e3001b25edea512f11" exitCode=0
Oct 30 19:14:28.611556 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.611526    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46" event={"ID":"4290d2e4-ccae-445f-b184-4509020ddefa","Type":"ContainerDied","Data":"5ebb28d2fe5028ff8ce1765c3c2d47f37101fc1b630a62e3001b25edea512f11"}
Oct 30 19:14:28.636113 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.625695    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:14:28.767452 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.767410    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4290d2e4-ccae-445f-b184-4509020ddefa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4290d2e4-ccae-445f-b184-4509020ddefa-istiod-ca-cert\") pod \"4290d2e4-ccae-445f-b184-4509020ddefa\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") "
Oct 30 19:14:28.768218 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.768106    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4290d2e4-ccae-445f-b184-4509020ddefa-istiod-ca-cert" pod "4290d2e4-ccae-445f-b184-4509020ddefa" (UID: "4290d2e4-ccae-445f-b184-4509020ddefa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:28.768810 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.768420    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-istio-envoy\") pod \"4290d2e4-ccae-445f-b184-4509020ddefa\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") "
Oct 30 19:14:28.769321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.768970    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4290d2e4-ccae-445f-b184-4509020ddefa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4290d2e4-ccae-445f-b184-4509020ddefa-istio-podinfo\") pod \"4290d2e4-ccae-445f-b184-4509020ddefa\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") "
Oct 30 19:14:28.769494 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.769478    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-workload-certs\") pod \"4290d2e4-ccae-445f-b184-4509020ddefa\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") "
Oct 30 19:14:28.769996 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.769935    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-workload-certs" pod "4290d2e4-ccae-445f-b184-4509020ddefa" (UID: "4290d2e4-ccae-445f-b184-4509020ddefa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:28.770717 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.770699    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4290d2e4-ccae-445f-b184-4509020ddefa-istio-token\" (UniqueName: \"kubernetes.io/projected/4290d2e4-ccae-445f-b184-4509020ddefa-istio-token\") pod \"4290d2e4-ccae-445f-b184-4509020ddefa\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") "
Oct 30 19:14:28.770954 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.770938    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-istio-data\") pod \"4290d2e4-ccae-445f-b184-4509020ddefa\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") "
Oct 30 19:14:28.771367 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.771350    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-workload-socket\") pod \"4290d2e4-ccae-445f-b184-4509020ddefa\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") "
Oct 30 19:14:28.771782 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.771767    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4290d2e4-ccae-445f-b184-4509020ddefa-kube-api-access-ssvkd\" (UniqueName: \"kubernetes.io/projected/4290d2e4-ccae-445f-b184-4509020ddefa-kube-api-access-ssvkd\") pod \"4290d2e4-ccae-445f-b184-4509020ddefa\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") "
Oct 30 19:14:28.772834 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.772816    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-credential-socket\") pod \"4290d2e4-ccae-445f-b184-4509020ddefa\" (UID: \"4290d2e4-ccae-445f-b184-4509020ddefa\") "
Oct 30 19:14:28.774360 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.774330    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4290d2e4-ccae-445f-b184-4509020ddefa-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:28.774494 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.774480    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:28.774568 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.771297    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-istio-data" pod "4290d2e4-ccae-445f-b184-4509020ddefa" (UID: "4290d2e4-ccae-445f-b184-4509020ddefa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:28.774640 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.771658    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-workload-socket" pod "4290d2e4-ccae-445f-b184-4509020ddefa" (UID: "4290d2e4-ccae-445f-b184-4509020ddefa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:28.774714 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.773180    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-credential-socket" pod "4290d2e4-ccae-445f-b184-4509020ddefa" (UID: "4290d2e4-ccae-445f-b184-4509020ddefa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:28.797198 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.779254    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4290d2e4-ccae-445f-b184-4509020ddefa-istio-token" pod "4290d2e4-ccae-445f-b184-4509020ddefa" (UID: "4290d2e4-ccae-445f-b184-4509020ddefa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:28.840187 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.828245    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4290d2e4-ccae-445f-b184-4509020ddefa-istio-podinfo" pod "4290d2e4-ccae-445f-b184-4509020ddefa" (UID: "4290d2e4-ccae-445f-b184-4509020ddefa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:28.872180 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.849299    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4290d2e4-ccae-445f-b184-4509020ddefa-kube-api-access-ssvkd" pod "4290d2e4-ccae-445f-b184-4509020ddefa" (UID: "4290d2e4-ccae-445f-b184-4509020ddefa"). InnerVolumeSpecName "kube-api-access-ssvkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:28.876321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.876210    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4290d2e4-ccae-445f-b184-4509020ddefa-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:28.876321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.876240    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4290d2e4-ccae-445f-b184-4509020ddefa-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:28.876321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.876256    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:28.876321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.876270    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:28.876321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.876284    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssvkd\" (UniqueName: \"kubernetes.io/projected/4290d2e4-ccae-445f-b184-4509020ddefa-kube-api-access-ssvkd\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:28.876321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.876298    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:28.971150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:28.956261    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-istio-envoy" pod "4290d2e4-ccae-445f-b184-4509020ddefa" (UID: "4290d2e4-ccae-445f-b184-4509020ddefa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:29.187865 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:29.182029    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4290d2e4-ccae-445f-b184-4509020ddefa-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:29.697171 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:29.695554    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46" event={"ID":"4290d2e4-ccae-445f-b184-4509020ddefa","Type":"ContainerDied","Data":"ef75dad68b067c40e35585795c6458786e69b379b752517082cde516fedd11ca"}
Oct 30 19:14:29.706858 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:29.706832    4372 scope.go:122] "RemoveContainer" containerID="5ebb28d2fe5028ff8ce1765c3c2d47f37101fc1b630a62e3001b25edea512f11"
Oct 30 19:14:29.712401 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:29.705448    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"
Oct 30 19:14:29.839114 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:29.839050    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"]
Oct 30 19:14:29.898211 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:29.897466    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-bpl46"]
Oct 30 19:14:30.283959 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:30.283927    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4290d2e4-ccae-445f-b184-4509020ddefa" path="/var/lib/kubelet/pods/4290d2e4-ccae-445f-b184-4509020ddefa/volumes"
Oct 30 19:14:30.913153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:30.912047    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"]
Oct 30 19:14:30.930034 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:30.927268    4372 cpu_manager.go:414] "containerMap: removing container" podUID="4290d2e4-ccae-445f-b184-4509020ddefa" containerName="istio-proxy"
Oct 30 19:14:30.930034 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:30.927297    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4290d2e4-ccae-445f-b184-4509020ddefa" containerName="istio-proxy"
Oct 30 19:14:30.930034 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:30.927321    4372 cpu_manager.go:414] "containerMap: removing container" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy"
Oct 30 19:14:30.930034 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:30.927332    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10111dcc-0436-4206-b798-d0bac119328a" containerName="istio-proxy"
Oct 30 19:14:30.930034 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:30.927590    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="4290d2e4-ccae-445f-b184-4509020ddefa" containerName="istio-proxy"
Oct 30 19:14:30.948221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:30.946656    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:30.957187 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:30.952173    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"agc-cred-9dc98f9b716b\""
Oct 30 19:14:30.957187 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:30.952320    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"payload-processing-external-model-e2e--3f19eb37-dockercfg-8qhnt\""
Oct 30 19:14:30.964722 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:30.964698    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"kube-root-ca.crt\""
Oct 30 19:14:30.965061 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:30.965034    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"payload-processing-external-model-plugins-e2e-extmod-a-21d896\""
Oct 30 19:14:30.965278 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:30.965249    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"openshift-service-ca.crt\""
Oct 30 19:14:30.990158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:30.969457    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"routing-overlay\""
Oct 30 19:14:30.990158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:30.988815    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"]
Oct 30 19:14:31.081343 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:31.076481    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6882bbe0-5cf3-4d63-868e-a9588b94bced-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:31.081681 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:31.081642    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6882bbe0-5cf3-4d63-868e-a9588b94bced-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:31.105149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:31.081978    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64cgb\" (UniqueName: \"kubernetes.io/projected/6882bbe0-5cf3-4d63-868e-a9588b94bced-kube-api-access-64cgb\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:31.105538 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:31.105507    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6882bbe0-5cf3-4d63-868e-a9588b94bced-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:31.259199 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:31.238221    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6882bbe0-5cf3-4d63-868e-a9588b94bced-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:31.263775 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:31.263740    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6882bbe0-5cf3-4d63-868e-a9588b94bced-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:31.265162 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:31.265114    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cgb\" (UniqueName: \"kubernetes.io/projected/6882bbe0-5cf3-4d63-868e-a9588b94bced-kube-api-access-64cgb\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:31.265774 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:31.265750    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6882bbe0-5cf3-4d63-868e-a9588b94bced-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:31.288776 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:31.280362    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6882bbe0-5cf3-4d63-868e-a9588b94bced-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:31.289577 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:31.264893    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6882bbe0-5cf3-4d63-868e-a9588b94bced-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:31.321173 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:31.306703    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6882bbe0-5cf3-4d63-868e-a9588b94bced-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:31.381824 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:31.381788    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64cgb\" (UniqueName: \"kubernetes.io/projected/6882bbe0-5cf3-4d63-868e-a9588b94bced-kube-api-access-64cgb\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:31.643770 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:31.643727    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:32.206542 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:14:32.206510    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6882bbe0_5cf3_4d63_868e_a9588b94bced.slice/crio-9ac339e64c926de478a247e7c1e9eb67a5db2f6149951a6e67765ec4b3e4dfcc WatchSource:0}: Error finding container 9ac339e64c926de478a247e7c1e9eb67a5db2f6149951a6e67765ec4b3e4dfcc: Status 404 returned error can't find the container with id 9ac339e64c926de478a247e7c1e9eb67a5db2f6149951a6e67765ec4b3e4dfcc
Oct 30 19:14:32.206688 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:32.206666    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"]
Oct 30 19:14:32.924510 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:32.924475    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7" event={"ID":"6882bbe0-5cf3-4d63-868e-a9588b94bced","Type":"ContainerStarted","Data":"a2b35bbc6f1a378d2a5a9d0c42392d9a58512b4b0b9d036995036929120d4cd2"}
Oct 30 19:14:32.924987 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:32.924525    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7" event={"ID":"6882bbe0-5cf3-4d63-868e-a9588b94bced","Type":"ContainerStarted","Data":"9ac339e64c926de478a247e7c1e9eb67a5db2f6149951a6e67765ec4b3e4dfcc"}
Oct 30 19:14:32.924987 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:32.924555    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:32.955974 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:32.955914    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7" podStartSLOduration=2.955893963 podStartE2EDuration="2.955893963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:32.949487351 +0000 UTC m=+2486.817905683" watchObservedRunningTime="2026-10-30 19:14:32.955893963 +0000 UTC m=+2486.824312387"
Oct 30 19:14:44.028410 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:14:44.028377    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:14:57.696972 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:14:57.696939    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:15:00.358234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.355252    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"]
Oct 30 19:15:00.363162 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.362342    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"
Oct 30 19:15:00.368199 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.368172    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"]
Oct 30 19:15:00.378935 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.378912    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"]
Oct 30 19:15:00.379058 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.379034    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"
Oct 30 19:15:00.380494 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.380456    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:15:00.381709 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.381694    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"]
Oct 30 19:15:00.386817 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.386795    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"
Oct 30 19:15:00.391468 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.388044    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"]
Oct 30 19:15:00.396267 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.395976    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"]
Oct 30 19:15:00.396267 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.396079    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"
Oct 30 19:15:00.412306 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.412276    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"]
Oct 30 19:15:00.418725 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.418696    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"]
Oct 30 19:15:00.428518 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.426754    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2vrs\" (UniqueName: \"kubernetes.io/projected/bc27b495-5bb4-4603-b302-14d42c209e91-kube-api-access-d2vrs\") pod \"maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w\" (UID: \"bc27b495-5bb4-4603-b302-14d42c209e91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"
Oct 30 19:15:00.428518 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.426867    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79318342-730a-44bc-9bb4-b1e4455ebc16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph\" (UID: \"79318342-730a-44bc-9bb4-b1e4455ebc16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"
Oct 30 19:15:00.428518 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.426977    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbsxn\" (UniqueName: \"kubernetes.io/projected/06d24a5b-5eae-49f0-ae25-4540bad7ade4-kube-api-access-kbsxn\") pod \"maas-api-key-cleanup-29889795-srkjf\" (UID: \"06d24a5b-5eae-49f0-ae25-4540bad7ade4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"
Oct 30 19:15:00.428518 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.427101    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc27b495-5bb4-4603-b302-14d42c209e91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w\" (UID: \"bc27b495-5bb4-4603-b302-14d42c209e91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"
Oct 30 19:15:00.428518 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.427213    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d24a5b-5eae-49f0-ae25-4540bad7ade4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889795-srkjf\" (UID: \"06d24a5b-5eae-49f0-ae25-4540bad7ade4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"
Oct 30 19:15:00.428518 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.427259    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hlzz\" (UniqueName: \"kubernetes.io/projected/79318342-730a-44bc-9bb4-b1e4455ebc16-kube-api-access-4hlzz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph\" (UID: \"79318342-730a-44bc-9bb4-b1e4455ebc16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"
Oct 30 19:15:00.429286 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.429271    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"]
Oct 30 19:15:00.429582 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.429389    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"
Oct 30 19:15:00.438445 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.438315    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"]
Oct 30 19:15:00.438556 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.438471    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"]
Oct 30 19:15:00.438706 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.438688    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"
Oct 30 19:15:00.446210 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.446189    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"
Oct 30 19:15:00.465933 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.465847    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"]
Oct 30 19:15:00.472615 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.472587    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"]
Oct 30 19:15:00.481440 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.481411    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"]
Oct 30 19:15:00.520040 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.520018    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"]
Oct 30 19:15:00.524803 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.524785    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"
Oct 30 19:15:00.528642 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.528620    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d24a5b-5eae-49f0-ae25-4540bad7ade4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889795-srkjf\" (UID: \"06d24a5b-5eae-49f0-ae25-4540bad7ade4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"
Oct 30 19:15:00.528789 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.528663    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hlzz\" (UniqueName: \"kubernetes.io/projected/79318342-730a-44bc-9bb4-b1e4455ebc16-kube-api-access-4hlzz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph\" (UID: \"79318342-730a-44bc-9bb4-b1e4455ebc16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"
Oct 30 19:15:00.528848 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.528784    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb42d228-1932-4501-8d6b-4b12c3b964e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7\" (UID: \"fb42d228-1932-4501-8d6b-4b12c3b964e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"
Oct 30 19:15:00.528909 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.528848    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vrs\" (UniqueName: \"kubernetes.io/projected/bc27b495-5bb4-4603-b302-14d42c209e91-kube-api-access-d2vrs\") pod \"maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w\" (UID: \"bc27b495-5bb4-4603-b302-14d42c209e91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"
Oct 30 19:15:00.529366 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.529343    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79318342-730a-44bc-9bb4-b1e4455ebc16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph\" (UID: \"79318342-730a-44bc-9bb4-b1e4455ebc16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"
Oct 30 19:15:00.529636 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.529529    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d24a5b-5eae-49f0-ae25-4540bad7ade4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889795-srkjf\" (UID: \"06d24a5b-5eae-49f0-ae25-4540bad7ade4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"
Oct 30 19:15:00.530342 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.530313    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79318342-730a-44bc-9bb4-b1e4455ebc16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph\" (UID: \"79318342-730a-44bc-9bb4-b1e4455ebc16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"
Oct 30 19:15:00.530550 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.530520    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbsxn\" (UniqueName: \"kubernetes.io/projected/06d24a5b-5eae-49f0-ae25-4540bad7ade4-kube-api-access-kbsxn\") pod \"maas-api-key-cleanup-29889795-srkjf\" (UID: \"06d24a5b-5eae-49f0-ae25-4540bad7ade4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"
Oct 30 19:15:00.530657 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.530637    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc27b495-5bb4-4603-b302-14d42c209e91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w\" (UID: \"bc27b495-5bb4-4603-b302-14d42c209e91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"
Oct 30 19:15:00.530719 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.530681    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjhxk\" (UniqueName: \"kubernetes.io/projected/fb42d228-1932-4501-8d6b-4b12c3b964e0-kube-api-access-kjhxk\") pod \"maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7\" (UID: \"fb42d228-1932-4501-8d6b-4b12c3b964e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"
Oct 30 19:15:00.531529 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.531502    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc27b495-5bb4-4603-b302-14d42c209e91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w\" (UID: \"bc27b495-5bb4-4603-b302-14d42c209e91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"
Oct 30 19:15:00.542989 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.542969    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"]
Oct 30 19:15:00.576728 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.576703    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbsxn\" (UniqueName: \"kubernetes.io/projected/06d24a5b-5eae-49f0-ae25-4540bad7ade4-kube-api-access-kbsxn\") pod \"maas-api-key-cleanup-29889795-srkjf\" (UID: \"06d24a5b-5eae-49f0-ae25-4540bad7ade4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"
Oct 30 19:15:00.577472 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.577448    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hlzz\" (UniqueName: \"kubernetes.io/projected/79318342-730a-44bc-9bb4-b1e4455ebc16-kube-api-access-4hlzz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph\" (UID: \"79318342-730a-44bc-9bb4-b1e4455ebc16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"
Oct 30 19:15:00.590325 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.590299    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2vrs\" (UniqueName: \"kubernetes.io/projected/bc27b495-5bb4-4603-b302-14d42c209e91-kube-api-access-d2vrs\") pod \"maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w\" (UID: \"bc27b495-5bb4-4603-b302-14d42c209e91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"
Oct 30 19:15:00.631932 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.631873    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2fc2cfe-9426-44a2-8492-9ced609b156a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng\" (UID: \"e2fc2cfe-9426-44a2-8492-9ced609b156a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"
Oct 30 19:15:00.632031 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.631931    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpvhp\" (UniqueName: \"kubernetes.io/projected/c46095e1-0538-4377-adad-69535ea14d47-kube-api-access-fpvhp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj\" (UID: \"c46095e1-0538-4377-adad-69535ea14d47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"
Oct 30 19:15:00.632031 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.631990    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f546370d-4f7c-4db2-a58e-6efb9db578e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth\" (UID: \"f546370d-4f7c-4db2-a58e-6efb9db578e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"
Oct 30 19:15:00.632136 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.632087    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb42d228-1932-4501-8d6b-4b12c3b964e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7\" (UID: \"fb42d228-1932-4501-8d6b-4b12c3b964e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"
Oct 30 19:15:00.632185 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.632168    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ca763d-c700-46aa-931e-2c583feaa266-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g\" (UID: \"d1ca763d-c700-46aa-931e-2c583feaa266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"
Oct 30 19:15:00.632224 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.632202    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vddc\" (UniqueName: \"kubernetes.io/projected/d1ca763d-c700-46aa-931e-2c583feaa266-kube-api-access-8vddc\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g\" (UID: \"d1ca763d-c700-46aa-931e-2c583feaa266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"
Oct 30 19:15:00.632347 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.632330    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbt4p\" (UniqueName: \"kubernetes.io/projected/e2fc2cfe-9426-44a2-8492-9ced609b156a-kube-api-access-bbt4p\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng\" (UID: \"e2fc2cfe-9426-44a2-8492-9ced609b156a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"
Oct 30 19:15:00.632405 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.632391    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhxk\" (UniqueName: \"kubernetes.io/projected/fb42d228-1932-4501-8d6b-4b12c3b964e0-kube-api-access-kjhxk\") pod \"maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7\" (UID: \"fb42d228-1932-4501-8d6b-4b12c3b964e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"
Oct 30 19:15:00.632467 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.632412    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cjsh\" (UniqueName: \"kubernetes.io/projected/f546370d-4f7c-4db2-a58e-6efb9db578e3-kube-api-access-5cjsh\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth\" (UID: \"f546370d-4f7c-4db2-a58e-6efb9db578e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"
Oct 30 19:15:00.632467 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.632436    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c46095e1-0538-4377-adad-69535ea14d47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj\" (UID: \"c46095e1-0538-4377-adad-69535ea14d47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"
Oct 30 19:15:00.632710 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.632692    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb42d228-1932-4501-8d6b-4b12c3b964e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7\" (UID: \"fb42d228-1932-4501-8d6b-4b12c3b964e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"
Oct 30 19:15:00.656776 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.656750    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjhxk\" (UniqueName: \"kubernetes.io/projected/fb42d228-1932-4501-8d6b-4b12c3b964e0-kube-api-access-kjhxk\") pod \"maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7\" (UID: \"fb42d228-1932-4501-8d6b-4b12c3b964e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"
Oct 30 19:15:00.686369 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.686349    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"
Oct 30 19:15:00.720401 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.720374    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"
Oct 30 19:15:00.733591 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.733563    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbt4p\" (UniqueName: \"kubernetes.io/projected/e2fc2cfe-9426-44a2-8492-9ced609b156a-kube-api-access-bbt4p\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng\" (UID: \"e2fc2cfe-9426-44a2-8492-9ced609b156a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"
Oct 30 19:15:00.733730 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.733651    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cjsh\" (UniqueName: \"kubernetes.io/projected/f546370d-4f7c-4db2-a58e-6efb9db578e3-kube-api-access-5cjsh\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth\" (UID: \"f546370d-4f7c-4db2-a58e-6efb9db578e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"
Oct 30 19:15:00.733730 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.733679    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c46095e1-0538-4377-adad-69535ea14d47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj\" (UID: \"c46095e1-0538-4377-adad-69535ea14d47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"
Oct 30 19:15:00.733855 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.733744    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2fc2cfe-9426-44a2-8492-9ced609b156a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng\" (UID: \"e2fc2cfe-9426-44a2-8492-9ced609b156a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"
Oct 30 19:15:00.733855 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.733801    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpvhp\" (UniqueName: \"kubernetes.io/projected/c46095e1-0538-4377-adad-69535ea14d47-kube-api-access-fpvhp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj\" (UID: \"c46095e1-0538-4377-adad-69535ea14d47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"
Oct 30 19:15:00.733855 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.733820    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f546370d-4f7c-4db2-a58e-6efb9db578e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth\" (UID: \"f546370d-4f7c-4db2-a58e-6efb9db578e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"
Oct 30 19:15:00.734041 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.733931    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ca763d-c700-46aa-931e-2c583feaa266-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g\" (UID: \"d1ca763d-c700-46aa-931e-2c583feaa266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"
Oct 30 19:15:00.734041 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.733962    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vddc\" (UniqueName: \"kubernetes.io/projected/d1ca763d-c700-46aa-931e-2c583feaa266-kube-api-access-8vddc\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g\" (UID: \"d1ca763d-c700-46aa-931e-2c583feaa266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"
Oct 30 19:15:00.734524 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.734500    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c46095e1-0538-4377-adad-69535ea14d47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj\" (UID: \"c46095e1-0538-4377-adad-69535ea14d47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"
Oct 30 19:15:00.734815 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.734793    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f546370d-4f7c-4db2-a58e-6efb9db578e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth\" (UID: \"f546370d-4f7c-4db2-a58e-6efb9db578e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"
Oct 30 19:15:00.734939 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.734913    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2fc2cfe-9426-44a2-8492-9ced609b156a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng\" (UID: \"e2fc2cfe-9426-44a2-8492-9ced609b156a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"
Oct 30 19:15:00.735100 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.735080    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ca763d-c700-46aa-931e-2c583feaa266-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g\" (UID: \"d1ca763d-c700-46aa-931e-2c583feaa266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"
Oct 30 19:15:00.748555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.748527    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vddc\" (UniqueName: \"kubernetes.io/projected/d1ca763d-c700-46aa-931e-2c583feaa266-kube-api-access-8vddc\") pod \"maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g\" (UID: \"d1ca763d-c700-46aa-931e-2c583feaa266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"
Oct 30 19:15:00.749598 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.749409    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"
Oct 30 19:15:00.752081 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.752058    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbt4p\" (UniqueName: \"kubernetes.io/projected/e2fc2cfe-9426-44a2-8492-9ced609b156a-kube-api-access-bbt4p\") pod \"maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng\" (UID: \"e2fc2cfe-9426-44a2-8492-9ced609b156a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"
Oct 30 19:15:00.752232 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.752210    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cjsh\" (UniqueName: \"kubernetes.io/projected/f546370d-4f7c-4db2-a58e-6efb9db578e3-kube-api-access-5cjsh\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth\" (UID: \"f546370d-4f7c-4db2-a58e-6efb9db578e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"
Oct 30 19:15:00.752538 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.752273    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpvhp\" (UniqueName: \"kubernetes.io/projected/c46095e1-0538-4377-adad-69535ea14d47-kube-api-access-fpvhp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj\" (UID: \"c46095e1-0538-4377-adad-69535ea14d47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"
Oct 30 19:15:00.752538 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.752297    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"
Oct 30 19:15:00.798544 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.798501    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"
Oct 30 19:15:00.800007 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.799975    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"
Oct 30 19:15:00.800151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.800118    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"
Oct 30 19:15:00.843435 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.842345    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"
Oct 30 19:15:00.993322 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:00.993266    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"]
Oct 30 19:15:00.998866 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:15:00.998662    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb42d228_1932_4501_8d6b_4b12c3b964e0.slice/crio-09505ac8e1dfbc0f336c8bb89842f6359efcc32e708d35fb6895424eeb04c0aa WatchSource:0}: Error finding container 09505ac8e1dfbc0f336c8bb89842f6359efcc32e708d35fb6895424eeb04c0aa: Status 404 returned error can't find the container with id 09505ac8e1dfbc0f336c8bb89842f6359efcc32e708d35fb6895424eeb04c0aa
Oct 30 19:15:01.245871 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:15:01.245840    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc27b495_5bb4_4603_b302_14d42c209e91.slice/crio-950bed5fa1be9b6276ecc12389856b4cc9349e4aad6ed0f618bb6d8e8aa98d2d WatchSource:0}: Error finding container 950bed5fa1be9b6276ecc12389856b4cc9349e4aad6ed0f618bb6d8e8aa98d2d: Status 404 returned error can't find the container with id 950bed5fa1be9b6276ecc12389856b4cc9349e4aad6ed0f618bb6d8e8aa98d2d
Oct 30 19:15:01.245985 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:01.245866    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"]
Oct 30 19:15:01.259347 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:01.259324    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"]
Oct 30 19:15:01.260261 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:15:01.260235    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46095e1_0538_4377_adad_69535ea14d47.slice/crio-c2e8cd4dd836b070adf46fd357d460096aa758ed8ac800286a7bf5cc3b3c1513 WatchSource:0}: Error finding container c2e8cd4dd836b070adf46fd357d460096aa758ed8ac800286a7bf5cc3b3c1513: Status 404 returned error can't find the container with id c2e8cd4dd836b070adf46fd357d460096aa758ed8ac800286a7bf5cc3b3c1513
Oct 30 19:15:01.262453 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:15:01.262244    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf546370d_4f7c_4db2_a58e_6efb9db578e3.slice/crio-64eeefbf621d87b736f718c8a53fb58bde4583ec49e4dd214544aba8a37d038c WatchSource:0}: Error finding container 64eeefbf621d87b736f718c8a53fb58bde4583ec49e4dd214544aba8a37d038c: Status 404 returned error can't find the container with id 64eeefbf621d87b736f718c8a53fb58bde4583ec49e4dd214544aba8a37d038c
Oct 30 19:15:01.265352 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:01.265332    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"]
Oct 30 19:15:01.297158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:01.296928    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"]
Oct 30 19:15:01.299363 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:15:01.298250    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79318342_730a_44bc_9bb4_b1e4455ebc16.slice/crio-03bbb5d3b02e56e24dd3ef0881472568696b5797e515539e2ccf98abca871458 WatchSource:0}: Error finding container 03bbb5d3b02e56e24dd3ef0881472568696b5797e515539e2ccf98abca871458: Status 404 returned error can't find the container with id 03bbb5d3b02e56e24dd3ef0881472568696b5797e515539e2ccf98abca871458
Oct 30 19:15:01.321504 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:15:01.321478    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d24a5b_5eae_49f0_ae25_4540bad7ade4.slice/crio-d71475e4c92fa1e75a7022b1f5ac2a03f36ea2e6272f50c89c5ffada51876e22 WatchSource:0}: Error finding container d71475e4c92fa1e75a7022b1f5ac2a03f36ea2e6272f50c89c5ffada51876e22: Status 404 returned error can't find the container with id d71475e4c92fa1e75a7022b1f5ac2a03f36ea2e6272f50c89c5ffada51876e22
Oct 30 19:15:01.323120 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:01.323097    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"]
Oct 30 19:15:01.362459 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:01.362428    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth" event={"ID":"f546370d-4f7c-4db2-a58e-6efb9db578e3","Type":"ContainerStarted","Data":"64eeefbf621d87b736f718c8a53fb58bde4583ec49e4dd214544aba8a37d038c"}
Oct 30 19:15:01.363766 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:01.363744    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj" event={"ID":"c46095e1-0538-4377-adad-69535ea14d47","Type":"ContainerStarted","Data":"c2e8cd4dd836b070adf46fd357d460096aa758ed8ac800286a7bf5cc3b3c1513"}
Oct 30 19:15:01.364907 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:01.364883    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7" event={"ID":"fb42d228-1932-4501-8d6b-4b12c3b964e0","Type":"ContainerStarted","Data":"09505ac8e1dfbc0f336c8bb89842f6359efcc32e708d35fb6895424eeb04c0aa"}
Oct 30 19:15:01.366166 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:01.366145    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf" event={"ID":"06d24a5b-5eae-49f0-ae25-4540bad7ade4","Type":"ContainerStarted","Data":"d71475e4c92fa1e75a7022b1f5ac2a03f36ea2e6272f50c89c5ffada51876e22"}
Oct 30 19:15:01.367662 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:01.367639    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph" event={"ID":"79318342-730a-44bc-9bb4-b1e4455ebc16","Type":"ContainerStarted","Data":"03bbb5d3b02e56e24dd3ef0881472568696b5797e515539e2ccf98abca871458"}
Oct 30 19:15:01.369301 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:01.369275    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w" event={"ID":"bc27b495-5bb4-4603-b302-14d42c209e91","Type":"ContainerStarted","Data":"950bed5fa1be9b6276ecc12389856b4cc9349e4aad6ed0f618bb6d8e8aa98d2d"}
Oct 30 19:15:01.391095 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:01.391067    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"]
Oct 30 19:15:01.410816 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:01.410795    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"]
Oct 30 19:15:01.549421 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:15:01.549390    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2fc2cfe_9426_44a2_8492_9ced609b156a.slice/crio-e51ad53f317e86c1479d1f785ee2db7a8057eff84b20fc79144b3d5a0dec11fa WatchSource:0}: Error finding container e51ad53f317e86c1479d1f785ee2db7a8057eff84b20fc79144b3d5a0dec11fa: Status 404 returned error can't find the container with id e51ad53f317e86c1479d1f785ee2db7a8057eff84b20fc79144b3d5a0dec11fa
Oct 30 19:15:02.266831 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:02.266706    4372 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb42d228_1932_4501_8d6b_4b12c3b964e0.slice/crio-c5e1d7eda1b7e650ec10fd581540f75197f0ea747d3877b4dcdfc901347ba106.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:15:02.459076 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.458683    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj" event={"ID":"c46095e1-0538-4377-adad-69535ea14d47","Type":"ContainerStarted","Data":"f243409f01e550ded81397da490f556bbd93c53a925be65803019db953e032dd"}
Oct 30 19:15:02.460708 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.460677    4372 generic.go:356] "Generic (PLEG): container finished" podID="fb42d228-1932-4501-8d6b-4b12c3b964e0" containerID="c5e1d7eda1b7e650ec10fd581540f75197f0ea747d3877b4dcdfc901347ba106" exitCode=60
Oct 30 19:15:02.460875 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.460762    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7" event={"ID":"fb42d228-1932-4501-8d6b-4b12c3b964e0","Type":"ContainerDied","Data":"c5e1d7eda1b7e650ec10fd581540f75197f0ea747d3877b4dcdfc901347ba106"}
Oct 30 19:15:02.461036 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.461020    4372 scope.go:122] "RemoveContainer" containerID="c5e1d7eda1b7e650ec10fd581540f75197f0ea747d3877b4dcdfc901347ba106"
Oct 30 19:15:02.463074 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.463048    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf" event={"ID":"06d24a5b-5eae-49f0-ae25-4540bad7ade4","Type":"ContainerStarted","Data":"6e7c2693aa54ac2b0ebe31f7e50324803bdfbb0d5fa23e0968ed2beb49c6fdd8"}
Oct 30 19:15:02.465515 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.465493    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph" event={"ID":"79318342-730a-44bc-9bb4-b1e4455ebc16","Type":"ContainerStarted","Data":"cbc5cb558f83ade7592bef9807ed8129668c7773c801a3b77a62b68b0f6e7200"}
Oct 30 19:15:02.467385 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.467362    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng" event={"ID":"e2fc2cfe-9426-44a2-8492-9ced609b156a","Type":"ContainerStarted","Data":"b7e11b984fd3b3b64dd40174ef7aa4c3ebbd7e4c13aab0017f0da068565e46d1"}
Oct 30 19:15:02.467499 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.467412    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng" event={"ID":"e2fc2cfe-9426-44a2-8492-9ced609b156a","Type":"ContainerStarted","Data":"e51ad53f317e86c1479d1f785ee2db7a8057eff84b20fc79144b3d5a0dec11fa"}
Oct 30 19:15:02.469056 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.469035    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w" event={"ID":"bc27b495-5bb4-4603-b302-14d42c209e91","Type":"ContainerStarted","Data":"ed1a5b7755eaa51a303577d76a8d1232c552045f913c05820abe05e51bf90831"}
Oct 30 19:15:02.470599 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.470576    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g" event={"ID":"d1ca763d-c700-46aa-931e-2c583feaa266","Type":"ContainerStarted","Data":"51ae9c76e3893c8f6d306364b46f5e487a936f6551eb1004a927cf8bf52cacfe"}
Oct 30 19:15:02.470707 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.470618    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g" event={"ID":"d1ca763d-c700-46aa-931e-2c583feaa266","Type":"ContainerStarted","Data":"681a7a290f4332e5fb1c030d267dedd27e859a03144292a55d7344925702361b"}
Oct 30 19:15:02.472088 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.472068    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth" event={"ID":"f546370d-4f7c-4db2-a58e-6efb9db578e3","Type":"ContainerStarted","Data":"41835d15bba5ec820364a72328f59ab3e4a53f6161d4014ff9275de854d5f341"}
Oct 30 19:15:02.494297 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.494243    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj" podStartSLOduration=2.494226011 podStartE2EDuration="2.494226011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.484947822 +0000 UTC m=+2516.353366153" watchObservedRunningTime="2026-10-30 19:15:02.494226011 +0000 UTC m=+2516.362644342"
Oct 30 19:15:02.507856 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.507797    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth" podStartSLOduration=2.507781776 podStartE2EDuration="2.507781776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.505668696 +0000 UTC m=+2516.374087025" watchObservedRunningTime="2026-10-30 19:15:02.507781776 +0000 UTC m=+2516.376200107"
Oct 30 19:15:02.543250 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.543194    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w" podStartSLOduration=2.5431085639999997 podStartE2EDuration="2.543108564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.542363538 +0000 UTC m=+2516.410781893" watchObservedRunningTime="2026-10-30 19:15:02.543108564 +0000 UTC m=+2516.411526897"
Oct 30 19:15:02.563893 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.563852    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph" podStartSLOduration=2.5638407069999998 podStartE2EDuration="2.563840707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.560912609 +0000 UTC m=+2516.429330943" watchObservedRunningTime="2026-10-30 19:15:02.563840707 +0000 UTC m=+2516.432259068"
Oct 30 19:15:02.585111 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.583661    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf" podStartSLOduration=2.583648922 podStartE2EDuration="2.583648922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.579383426 +0000 UTC m=+2516.447801753" watchObservedRunningTime="2026-10-30 19:15:02.583648922 +0000 UTC m=+2516.452067252"
Oct 30 19:15:02.602903 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.602860    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng" podStartSLOduration=2.602847073 podStartE2EDuration="2.602847073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.597857925 +0000 UTC m=+2516.466276255" watchObservedRunningTime="2026-10-30 19:15:02.602847073 +0000 UTC m=+2516.471265403"
Oct 30 19:15:02.628526 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:02.628486    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g" podStartSLOduration=2.628474153 podStartE2EDuration="2.628474153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.623687473 +0000 UTC m=+2516.492105804" watchObservedRunningTime="2026-10-30 19:15:02.628474153 +0000 UTC m=+2516.496892512"
Oct 30 19:15:03.481847 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.481752    4372 generic.go:356] "Generic (PLEG): container finished" podID="c46095e1-0538-4377-adad-69535ea14d47" containerID="f243409f01e550ded81397da490f556bbd93c53a925be65803019db953e032dd" exitCode=60
Oct 30 19:15:03.482386 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.481834    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj" event={"ID":"c46095e1-0538-4377-adad-69535ea14d47","Type":"ContainerDied","Data":"f243409f01e550ded81397da490f556bbd93c53a925be65803019db953e032dd"}
Oct 30 19:15:03.482386 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.482289    4372 scope.go:122] "RemoveContainer" containerID="f243409f01e550ded81397da490f556bbd93c53a925be65803019db953e032dd"
Oct 30 19:15:03.484095 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.484061    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7" event={"ID":"fb42d228-1932-4501-8d6b-4b12c3b964e0","Type":"ContainerStarted","Data":"fd5c1dd38bcd5c148cdedb0de95a6d825ce90abd7c96850926356ddb120bb146"}
Oct 30 19:15:03.485810 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.485790    4372 generic.go:356] "Generic (PLEG): container finished" podID="06d24a5b-5eae-49f0-ae25-4540bad7ade4" containerID="6e7c2693aa54ac2b0ebe31f7e50324803bdfbb0d5fa23e0968ed2beb49c6fdd8" exitCode=60
Oct 30 19:15:03.485926 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.485881    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf" event={"ID":"06d24a5b-5eae-49f0-ae25-4540bad7ade4","Type":"ContainerDied","Data":"6e7c2693aa54ac2b0ebe31f7e50324803bdfbb0d5fa23e0968ed2beb49c6fdd8"}
Oct 30 19:15:03.486252 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.486230    4372 scope.go:122] "RemoveContainer" containerID="6e7c2693aa54ac2b0ebe31f7e50324803bdfbb0d5fa23e0968ed2beb49c6fdd8"
Oct 30 19:15:03.487751 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.487730    4372 generic.go:356] "Generic (PLEG): container finished" podID="79318342-730a-44bc-9bb4-b1e4455ebc16" containerID="cbc5cb558f83ade7592bef9807ed8129668c7773c801a3b77a62b68b0f6e7200" exitCode=60
Oct 30 19:15:03.487869 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.487794    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph" event={"ID":"79318342-730a-44bc-9bb4-b1e4455ebc16","Type":"ContainerDied","Data":"cbc5cb558f83ade7592bef9807ed8129668c7773c801a3b77a62b68b0f6e7200"}
Oct 30 19:15:03.488058 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.488040    4372 scope.go:122] "RemoveContainer" containerID="cbc5cb558f83ade7592bef9807ed8129668c7773c801a3b77a62b68b0f6e7200"
Oct 30 19:15:03.489560 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.489539    4372 generic.go:356] "Generic (PLEG): container finished" podID="e2fc2cfe-9426-44a2-8492-9ced609b156a" containerID="b7e11b984fd3b3b64dd40174ef7aa4c3ebbd7e4c13aab0017f0da068565e46d1" exitCode=60
Oct 30 19:15:03.489659 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.489614    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng" event={"ID":"e2fc2cfe-9426-44a2-8492-9ced609b156a","Type":"ContainerDied","Data":"b7e11b984fd3b3b64dd40174ef7aa4c3ebbd7e4c13aab0017f0da068565e46d1"}
Oct 30 19:15:03.489789 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.489776    4372 scope.go:122] "RemoveContainer" containerID="b7e11b984fd3b3b64dd40174ef7aa4c3ebbd7e4c13aab0017f0da068565e46d1"
Oct 30 19:15:03.491095 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.491075    4372 generic.go:356] "Generic (PLEG): container finished" podID="bc27b495-5bb4-4603-b302-14d42c209e91" containerID="ed1a5b7755eaa51a303577d76a8d1232c552045f913c05820abe05e51bf90831" exitCode=60
Oct 30 19:15:03.491202 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.491169    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w" event={"ID":"bc27b495-5bb4-4603-b302-14d42c209e91","Type":"ContainerDied","Data":"ed1a5b7755eaa51a303577d76a8d1232c552045f913c05820abe05e51bf90831"}
Oct 30 19:15:03.491419 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.491405    4372 scope.go:122] "RemoveContainer" containerID="ed1a5b7755eaa51a303577d76a8d1232c552045f913c05820abe05e51bf90831"
Oct 30 19:15:03.492739 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.492721    4372 generic.go:356] "Generic (PLEG): container finished" podID="d1ca763d-c700-46aa-931e-2c583feaa266" containerID="51ae9c76e3893c8f6d306364b46f5e487a936f6551eb1004a927cf8bf52cacfe" exitCode=60
Oct 30 19:15:03.492822 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.492802    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g" event={"ID":"d1ca763d-c700-46aa-931e-2c583feaa266","Type":"ContainerDied","Data":"51ae9c76e3893c8f6d306364b46f5e487a936f6551eb1004a927cf8bf52cacfe"}
Oct 30 19:15:03.492996 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.492974    4372 scope.go:122] "RemoveContainer" containerID="51ae9c76e3893c8f6d306364b46f5e487a936f6551eb1004a927cf8bf52cacfe"
Oct 30 19:15:03.494740 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.494715    4372 generic.go:356] "Generic (PLEG): container finished" podID="f546370d-4f7c-4db2-a58e-6efb9db578e3" containerID="41835d15bba5ec820364a72328f59ab3e4a53f6161d4014ff9275de854d5f341" exitCode=60
Oct 30 19:15:03.494852 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.494774    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth" event={"ID":"f546370d-4f7c-4db2-a58e-6efb9db578e3","Type":"ContainerDied","Data":"41835d15bba5ec820364a72328f59ab3e4a53f6161d4014ff9275de854d5f341"}
Oct 30 19:15:03.495342 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.495318    4372 scope.go:122] "RemoveContainer" containerID="41835d15bba5ec820364a72328f59ab3e4a53f6161d4014ff9275de854d5f341"
Oct 30 19:15:03.545481 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:03.545425    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7" podStartSLOduration=3.545393007 podStartE2EDuration="3.545393007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.540396425 +0000 UTC m=+2517.408814755" watchObservedRunningTime="2026-10-30 19:15:03.545393007 +0000 UTC m=+2517.413811340"
Oct 30 19:15:04.505002 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:04.504900    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf" event={"ID":"06d24a5b-5eae-49f0-ae25-4540bad7ade4","Type":"ContainerStarted","Data":"e06a0e01e7f265586d1ead6f18585195bb412490e3dcafae120efb06e5bc7318"}
Oct 30 19:15:04.517870 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:04.517837    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph" event={"ID":"79318342-730a-44bc-9bb4-b1e4455ebc16","Type":"ContainerStarted","Data":"43a879f169b3a8b87b14b77629ee8d1eb84effd4a5d34d998cdb75ba468a76a5"}
Oct 30 19:15:04.520501 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:04.520468    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng" event={"ID":"e2fc2cfe-9426-44a2-8492-9ced609b156a","Type":"ContainerStarted","Data":"0daabef96c7c181738ef162a587a2e4a5c4f92bb57c01a0f5391660aeafe70a5"}
Oct 30 19:15:04.522773 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:04.522745    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w" event={"ID":"bc27b495-5bb4-4603-b302-14d42c209e91","Type":"ContainerStarted","Data":"2338bb89faf1f68cc3784d6cf6dc4d7c1a4d8d4ec469b95de51aa361435d0901"}
Oct 30 19:15:04.525469 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:04.525445    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g" event={"ID":"d1ca763d-c700-46aa-931e-2c583feaa266","Type":"ContainerStarted","Data":"41d94aebb6eb452d054560594586fcfa097f394117fc0530f5112822857675e9"}
Oct 30 19:15:04.527385 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:04.527361    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth" event={"ID":"f546370d-4f7c-4db2-a58e-6efb9db578e3","Type":"ContainerStarted","Data":"e0866e28d3e7c242eeb2f5767fb9663e3aa6d7f01769e8e2f2dc85d7094e264e"}
Oct 30 19:15:04.529987 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:04.529955    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj" event={"ID":"c46095e1-0538-4377-adad-69535ea14d47","Type":"ContainerStarted","Data":"408637720ea69dc6c9bdf4e5129a745b687ec122ed758c88a2cb1e89ec04cc00"}
Oct 30 19:15:04.531942 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:04.531915    4372 generic.go:356] "Generic (PLEG): container finished" podID="fb42d228-1932-4501-8d6b-4b12c3b964e0" containerID="fd5c1dd38bcd5c148cdedb0de95a6d825ce90abd7c96850926356ddb120bb146" exitCode=60
Oct 30 19:15:04.532032 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:04.531952    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7" event={"ID":"fb42d228-1932-4501-8d6b-4b12c3b964e0","Type":"ContainerDied","Data":"fd5c1dd38bcd5c148cdedb0de95a6d825ce90abd7c96850926356ddb120bb146"}
Oct 30 19:15:04.532032 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:04.531995    4372 scope.go:122] "RemoveContainer" containerID="c5e1d7eda1b7e650ec10fd581540f75197f0ea747d3877b4dcdfc901347ba106"
Oct 30 19:15:04.532229 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:04.532212    4372 scope.go:122] "RemoveContainer" containerID="fd5c1dd38bcd5c148cdedb0de95a6d825ce90abd7c96850926356ddb120bb146"
Oct 30 19:15:04.532486 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:04.532466    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7_odh-ai-gateway-infra(fb42d228-1932-4501-8d6b-4b12c3b964e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7" podUID="fb42d228-1932-4501-8d6b-4b12c3b964e0"
Oct 30 19:15:05.542855 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.542824    4372 generic.go:356] "Generic (PLEG): container finished" podID="06d24a5b-5eae-49f0-ae25-4540bad7ade4" containerID="e06a0e01e7f265586d1ead6f18585195bb412490e3dcafae120efb06e5bc7318" exitCode=60
Oct 30 19:15:05.543360 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.542861    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf" event={"ID":"06d24a5b-5eae-49f0-ae25-4540bad7ade4","Type":"ContainerDied","Data":"e06a0e01e7f265586d1ead6f18585195bb412490e3dcafae120efb06e5bc7318"}
Oct 30 19:15:05.543360 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.542936    4372 scope.go:122] "RemoveContainer" containerID="6e7c2693aa54ac2b0ebe31f7e50324803bdfbb0d5fa23e0968ed2beb49c6fdd8"
Oct 30 19:15:05.543360 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.543251    4372 scope.go:122] "RemoveContainer" containerID="e06a0e01e7f265586d1ead6f18585195bb412490e3dcafae120efb06e5bc7318"
Oct 30 19:15:05.543583 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:05.543569    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889795-srkjf_odh-ai-gateway-infra(06d24a5b-5eae-49f0-ae25-4540bad7ade4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf" podUID="06d24a5b-5eae-49f0-ae25-4540bad7ade4"
Oct 30 19:15:05.545183 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.545160    4372 generic.go:356] "Generic (PLEG): container finished" podID="79318342-730a-44bc-9bb4-b1e4455ebc16" containerID="43a879f169b3a8b87b14b77629ee8d1eb84effd4a5d34d998cdb75ba468a76a5" exitCode=60
Oct 30 19:15:05.545330 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.545237    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph" event={"ID":"79318342-730a-44bc-9bb4-b1e4455ebc16","Type":"ContainerDied","Data":"43a879f169b3a8b87b14b77629ee8d1eb84effd4a5d34d998cdb75ba468a76a5"}
Oct 30 19:15:05.545455 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.545438    4372 scope.go:122] "RemoveContainer" containerID="43a879f169b3a8b87b14b77629ee8d1eb84effd4a5d34d998cdb75ba468a76a5"
Oct 30 19:15:05.545745 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:05.545724    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph_odh-ai-gateway-infra(79318342-730a-44bc-9bb4-b1e4455ebc16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph" podUID="79318342-730a-44bc-9bb4-b1e4455ebc16"
Oct 30 19:15:05.547262 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.547238    4372 generic.go:356] "Generic (PLEG): container finished" podID="e2fc2cfe-9426-44a2-8492-9ced609b156a" containerID="0daabef96c7c181738ef162a587a2e4a5c4f92bb57c01a0f5391660aeafe70a5" exitCode=60
Oct 30 19:15:05.547383 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.547312    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng" event={"ID":"e2fc2cfe-9426-44a2-8492-9ced609b156a","Type":"ContainerDied","Data":"0daabef96c7c181738ef162a587a2e4a5c4f92bb57c01a0f5391660aeafe70a5"}
Oct 30 19:15:05.547565 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.547545    4372 scope.go:122] "RemoveContainer" containerID="0daabef96c7c181738ef162a587a2e4a5c4f92bb57c01a0f5391660aeafe70a5"
Oct 30 19:15:05.547840 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:05.547816    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng_odh-ai-gateway-infra(e2fc2cfe-9426-44a2-8492-9ced609b156a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng" podUID="e2fc2cfe-9426-44a2-8492-9ced609b156a"
Oct 30 19:15:05.549425 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.549407    4372 generic.go:356] "Generic (PLEG): container finished" podID="bc27b495-5bb4-4603-b302-14d42c209e91" containerID="2338bb89faf1f68cc3784d6cf6dc4d7c1a4d8d4ec469b95de51aa361435d0901" exitCode=60
Oct 30 19:15:05.549505 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.549476    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w" event={"ID":"bc27b495-5bb4-4603-b302-14d42c209e91","Type":"ContainerDied","Data":"2338bb89faf1f68cc3784d6cf6dc4d7c1a4d8d4ec469b95de51aa361435d0901"}
Oct 30 19:15:05.549714 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.549699    4372 scope.go:122] "RemoveContainer" containerID="2338bb89faf1f68cc3784d6cf6dc4d7c1a4d8d4ec469b95de51aa361435d0901"
Oct 30 19:15:05.549997 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:05.549976    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w_odh-ai-gateway-infra(bc27b495-5bb4-4603-b302-14d42c209e91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w" podUID="bc27b495-5bb4-4603-b302-14d42c209e91"
Oct 30 19:15:05.551717 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.551692    4372 generic.go:356] "Generic (PLEG): container finished" podID="d1ca763d-c700-46aa-931e-2c583feaa266" containerID="41d94aebb6eb452d054560594586fcfa097f394117fc0530f5112822857675e9" exitCode=60
Oct 30 19:15:05.551900 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.551882    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g" event={"ID":"d1ca763d-c700-46aa-931e-2c583feaa266","Type":"ContainerDied","Data":"41d94aebb6eb452d054560594586fcfa097f394117fc0530f5112822857675e9"}
Oct 30 19:15:05.552248 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.552233    4372 scope.go:122] "RemoveContainer" containerID="41d94aebb6eb452d054560594586fcfa097f394117fc0530f5112822857675e9"
Oct 30 19:15:05.552488 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:05.552470    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g_odh-ai-gateway-infra(d1ca763d-c700-46aa-931e-2c583feaa266)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g" podUID="d1ca763d-c700-46aa-931e-2c583feaa266"
Oct 30 19:15:05.554639 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.554617    4372 generic.go:356] "Generic (PLEG): container finished" podID="f546370d-4f7c-4db2-a58e-6efb9db578e3" containerID="e0866e28d3e7c242eeb2f5767fb9663e3aa6d7f01769e8e2f2dc85d7094e264e" exitCode=60
Oct 30 19:15:05.554781 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.554761    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth" event={"ID":"f546370d-4f7c-4db2-a58e-6efb9db578e3","Type":"ContainerDied","Data":"e0866e28d3e7c242eeb2f5767fb9663e3aa6d7f01769e8e2f2dc85d7094e264e"}
Oct 30 19:15:05.555179 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.555157    4372 scope.go:122] "RemoveContainer" containerID="e0866e28d3e7c242eeb2f5767fb9663e3aa6d7f01769e8e2f2dc85d7094e264e"
Oct 30 19:15:05.555483 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:05.555457    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth_odh-ai-gateway-infra(f546370d-4f7c-4db2-a58e-6efb9db578e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth" podUID="f546370d-4f7c-4db2-a58e-6efb9db578e3"
Oct 30 19:15:05.557119 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.557097    4372 generic.go:356] "Generic (PLEG): container finished" podID="c46095e1-0538-4377-adad-69535ea14d47" containerID="408637720ea69dc6c9bdf4e5129a745b687ec122ed758c88a2cb1e89ec04cc00" exitCode=60
Oct 30 19:15:05.557255 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.557173    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj" event={"ID":"c46095e1-0538-4377-adad-69535ea14d47","Type":"ContainerDied","Data":"408637720ea69dc6c9bdf4e5129a745b687ec122ed758c88a2cb1e89ec04cc00"}
Oct 30 19:15:05.557410 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.557395    4372 scope.go:122] "RemoveContainer" containerID="408637720ea69dc6c9bdf4e5129a745b687ec122ed758c88a2cb1e89ec04cc00"
Oct 30 19:15:05.557711 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:05.557690    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj_odh-ai-gateway-infra(c46095e1-0538-4377-adad-69535ea14d47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj" podUID="c46095e1-0538-4377-adad-69535ea14d47"
Oct 30 19:15:05.559316 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.559216    4372 scope.go:122] "RemoveContainer" containerID="fd5c1dd38bcd5c148cdedb0de95a6d825ce90abd7c96850926356ddb120bb146"
Oct 30 19:15:05.559387 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.559348    4372 scope.go:122] "RemoveContainer" containerID="cbc5cb558f83ade7592bef9807ed8129668c7773c801a3b77a62b68b0f6e7200"
Oct 30 19:15:05.559504 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:05.559483    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7_odh-ai-gateway-infra(fb42d228-1932-4501-8d6b-4b12c3b964e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7" podUID="fb42d228-1932-4501-8d6b-4b12c3b964e0"
Oct 30 19:15:05.573674 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.573657    4372 scope.go:122] "RemoveContainer" containerID="b7e11b984fd3b3b64dd40174ef7aa4c3ebbd7e4c13aab0017f0da068565e46d1"
Oct 30 19:15:05.589439 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.589420    4372 scope.go:122] "RemoveContainer" containerID="ed1a5b7755eaa51a303577d76a8d1232c552045f913c05820abe05e51bf90831"
Oct 30 19:15:05.603602 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.603582    4372 scope.go:122] "RemoveContainer" containerID="51ae9c76e3893c8f6d306364b46f5e487a936f6551eb1004a927cf8bf52cacfe"
Oct 30 19:15:05.617318 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.617300    4372 scope.go:122] "RemoveContainer" containerID="41835d15bba5ec820364a72328f59ab3e4a53f6161d4014ff9275de854d5f341"
Oct 30 19:15:05.630887 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:05.630846    4372 scope.go:122] "RemoveContainer" containerID="f243409f01e550ded81397da490f556bbd93c53a925be65803019db953e032dd"
Oct 30 19:15:06.568896 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:06.568872    4372 scope.go:122] "RemoveContainer" containerID="43a879f169b3a8b87b14b77629ee8d1eb84effd4a5d34d998cdb75ba468a76a5"
Oct 30 19:15:06.569403 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:06.569168    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph_odh-ai-gateway-infra(79318342-730a-44bc-9bb4-b1e4455ebc16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph" podUID="79318342-730a-44bc-9bb4-b1e4455ebc16"
Oct 30 19:15:06.570609 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:06.570591    4372 scope.go:122] "RemoveContainer" containerID="0daabef96c7c181738ef162a587a2e4a5c4f92bb57c01a0f5391660aeafe70a5"
Oct 30 19:15:06.570872 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:06.570851    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng_odh-ai-gateway-infra(e2fc2cfe-9426-44a2-8492-9ced609b156a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng" podUID="e2fc2cfe-9426-44a2-8492-9ced609b156a"
Oct 30 19:15:06.572323 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:06.572306    4372 scope.go:122] "RemoveContainer" containerID="2338bb89faf1f68cc3784d6cf6dc4d7c1a4d8d4ec469b95de51aa361435d0901"
Oct 30 19:15:06.572575 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:06.572549    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w_odh-ai-gateway-infra(bc27b495-5bb4-4603-b302-14d42c209e91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w" podUID="bc27b495-5bb4-4603-b302-14d42c209e91"
Oct 30 19:15:06.573927 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:06.573908    4372 scope.go:122] "RemoveContainer" containerID="41d94aebb6eb452d054560594586fcfa097f394117fc0530f5112822857675e9"
Oct 30 19:15:06.574189 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:06.574169    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g_odh-ai-gateway-infra(d1ca763d-c700-46aa-931e-2c583feaa266)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g" podUID="d1ca763d-c700-46aa-931e-2c583feaa266"
Oct 30 19:15:06.575603 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:06.575575    4372 scope.go:122] "RemoveContainer" containerID="e0866e28d3e7c242eeb2f5767fb9663e3aa6d7f01769e8e2f2dc85d7094e264e"
Oct 30 19:15:06.575824 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:06.575804    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth_odh-ai-gateway-infra(f546370d-4f7c-4db2-a58e-6efb9db578e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth" podUID="f546370d-4f7c-4db2-a58e-6efb9db578e3"
Oct 30 19:15:06.577318 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:06.577296    4372 scope.go:122] "RemoveContainer" containerID="408637720ea69dc6c9bdf4e5129a745b687ec122ed758c88a2cb1e89ec04cc00"
Oct 30 19:15:06.577585 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:06.577562    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj_odh-ai-gateway-infra(c46095e1-0538-4377-adad-69535ea14d47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj" podUID="c46095e1-0538-4377-adad-69535ea14d47"
Oct 30 19:15:06.578843 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:06.578823    4372 scope.go:122] "RemoveContainer" containerID="e06a0e01e7f265586d1ead6f18585195bb412490e3dcafae120efb06e5bc7318"
Oct 30 19:15:06.579085 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:06.579066    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889795-srkjf_odh-ai-gateway-infra(06d24a5b-5eae-49f0-ae25-4540bad7ade4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf" podUID="06d24a5b-5eae-49f0-ae25-4540bad7ade4"
Oct 30 19:15:15.258776 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:15.258742    4372 scope.go:122] "RemoveContainer" containerID="e0866e28d3e7c242eeb2f5767fb9663e3aa6d7f01769e8e2f2dc85d7094e264e"
Oct 30 19:15:16.264083 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:16.264051    4372 scope.go:122] "RemoveContainer" containerID="43a879f169b3a8b87b14b77629ee8d1eb84effd4a5d34d998cdb75ba468a76a5"
Oct 30 19:15:16.679823 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:16.679793    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph" event={"ID":"79318342-730a-44bc-9bb4-b1e4455ebc16","Type":"ContainerStarted","Data":"ec22470f7585ba96c7bafba6a5d44a7a26c0d468cd54531f38070764b01682c5"}
Oct 30 19:15:16.681960 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:16.681935    4372 generic.go:356] "Generic (PLEG): container finished" podID="f546370d-4f7c-4db2-a58e-6efb9db578e3" containerID="00e6a6c8359aa1e1c8cdfcb8c84c5d6379aa7a2c3395e24b74c43187bb1c99c6" exitCode=60
Oct 30 19:15:16.682069 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:16.682010    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth" event={"ID":"f546370d-4f7c-4db2-a58e-6efb9db578e3","Type":"ContainerDied","Data":"00e6a6c8359aa1e1c8cdfcb8c84c5d6379aa7a2c3395e24b74c43187bb1c99c6"}
Oct 30 19:15:16.682069 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:16.682066    4372 scope.go:122] "RemoveContainer" containerID="e0866e28d3e7c242eeb2f5767fb9663e3aa6d7f01769e8e2f2dc85d7094e264e"
Oct 30 19:15:16.682402 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:16.682384    4372 scope.go:122] "RemoveContainer" containerID="00e6a6c8359aa1e1c8cdfcb8c84c5d6379aa7a2c3395e24b74c43187bb1c99c6"
Oct 30 19:15:16.682672 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:16.682651    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth_odh-ai-gateway-infra(f546370d-4f7c-4db2-a58e-6efb9db578e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth" podUID="f546370d-4f7c-4db2-a58e-6efb9db578e3"
Oct 30 19:15:17.259186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:17.259160    4372 scope.go:122] "RemoveContainer" containerID="41d94aebb6eb452d054560594586fcfa097f394117fc0530f5112822857675e9"
Oct 30 19:15:17.259327 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:17.259299    4372 scope.go:122] "RemoveContainer" containerID="e06a0e01e7f265586d1ead6f18585195bb412490e3dcafae120efb06e5bc7318"
Oct 30 19:15:17.259401 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:17.259326    4372 scope.go:122] "RemoveContainer" containerID="2338bb89faf1f68cc3784d6cf6dc4d7c1a4d8d4ec469b95de51aa361435d0901"
Oct 30 19:15:17.259478 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:17.259458    4372 scope.go:122] "RemoveContainer" containerID="fd5c1dd38bcd5c148cdedb0de95a6d825ce90abd7c96850926356ddb120bb146"
Oct 30 19:15:17.696357 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:17.696328    4372 generic.go:356] "Generic (PLEG): container finished" podID="79318342-730a-44bc-9bb4-b1e4455ebc16" containerID="ec22470f7585ba96c7bafba6a5d44a7a26c0d468cd54531f38070764b01682c5" exitCode=60
Oct 30 19:15:17.696802 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:17.696360    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph" event={"ID":"79318342-730a-44bc-9bb4-b1e4455ebc16","Type":"ContainerDied","Data":"ec22470f7585ba96c7bafba6a5d44a7a26c0d468cd54531f38070764b01682c5"}
Oct 30 19:15:17.696802 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:17.696423    4372 scope.go:122] "RemoveContainer" containerID="43a879f169b3a8b87b14b77629ee8d1eb84effd4a5d34d998cdb75ba468a76a5"
Oct 30 19:15:17.696987 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:17.696966    4372 scope.go:122] "RemoveContainer" containerID="ec22470f7585ba96c7bafba6a5d44a7a26c0d468cd54531f38070764b01682c5"
Oct 30 19:15:17.697373 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:17.697349    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph_odh-ai-gateway-infra(79318342-730a-44bc-9bb4-b1e4455ebc16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph" podUID="79318342-730a-44bc-9bb4-b1e4455ebc16"
Oct 30 19:15:17.712073 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:17.712055    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"]
Oct 30 19:15:17.740236 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:17.740215    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"]
Oct 30 19:15:18.009934 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.009910    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"
Oct 30 19:15:18.061260 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.061188    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f546370d-4f7c-4db2-a58e-6efb9db578e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f546370d-4f7c-4db2-a58e-6efb9db578e3-maas-api-service-ca\") pod \"f546370d-4f7c-4db2-a58e-6efb9db578e3\" (UID: \"f546370d-4f7c-4db2-a58e-6efb9db578e3\") "
Oct 30 19:15:18.061429 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.061377    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f546370d-4f7c-4db2-a58e-6efb9db578e3-kube-api-access-5cjsh\" (UniqueName: \"kubernetes.io/projected/f546370d-4f7c-4db2-a58e-6efb9db578e3-kube-api-access-5cjsh\") pod \"f546370d-4f7c-4db2-a58e-6efb9db578e3\" (UID: \"f546370d-4f7c-4db2-a58e-6efb9db578e3\") "
Oct 30 19:15:18.061579 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.061556    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f546370d-4f7c-4db2-a58e-6efb9db578e3-maas-api-service-ca" pod "f546370d-4f7c-4db2-a58e-6efb9db578e3" (UID: "f546370d-4f7c-4db2-a58e-6efb9db578e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:18.062069 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.062046    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f546370d-4f7c-4db2-a58e-6efb9db578e3-maas-api-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:18.063862 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.063828    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f546370d-4f7c-4db2-a58e-6efb9db578e3-kube-api-access-5cjsh" pod "f546370d-4f7c-4db2-a58e-6efb9db578e3" (UID: "f546370d-4f7c-4db2-a58e-6efb9db578e3"). InnerVolumeSpecName "kube-api-access-5cjsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:18.164243 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.164212    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cjsh\" (UniqueName: \"kubernetes.io/projected/f546370d-4f7c-4db2-a58e-6efb9db578e3-kube-api-access-5cjsh\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:18.258253 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.258228    4372 scope.go:122] "RemoveContainer" containerID="0daabef96c7c181738ef162a587a2e4a5c4f92bb57c01a0f5391660aeafe70a5"
Oct 30 19:15:18.710640 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.710613    4372 generic.go:356] "Generic (PLEG): container finished" podID="06d24a5b-5eae-49f0-ae25-4540bad7ade4" containerID="ee2b63852ce8c5b9f2c97f539d769a654a405d761768bca259a095f21ac5aa48" exitCode=60
Oct 30 19:15:18.711000 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.710682    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf" event={"ID":"06d24a5b-5eae-49f0-ae25-4540bad7ade4","Type":"ContainerDied","Data":"ee2b63852ce8c5b9f2c97f539d769a654a405d761768bca259a095f21ac5aa48"}
Oct 30 19:15:18.711000 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.710745    4372 scope.go:122] "RemoveContainer" containerID="e06a0e01e7f265586d1ead6f18585195bb412490e3dcafae120efb06e5bc7318"
Oct 30 19:15:18.711107 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.711078    4372 scope.go:122] "RemoveContainer" containerID="ee2b63852ce8c5b9f2c97f539d769a654a405d761768bca259a095f21ac5aa48"
Oct 30 19:15:18.711461 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:18.711436    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889795-srkjf_odh-ai-gateway-infra(06d24a5b-5eae-49f0-ae25-4540bad7ade4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf" podUID="06d24a5b-5eae-49f0-ae25-4540bad7ade4"
Oct 30 19:15:18.714996 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.714959    4372 generic.go:356] "Generic (PLEG): container finished" podID="bc27b495-5bb4-4603-b302-14d42c209e91" containerID="9147583df277cc6d77badc7527a7cb2dff9bc6db450c17fc972ed1ff16b2ab76" exitCode=60
Oct 30 19:15:18.715206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.715034    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w" event={"ID":"bc27b495-5bb4-4603-b302-14d42c209e91","Type":"ContainerDied","Data":"9147583df277cc6d77badc7527a7cb2dff9bc6db450c17fc972ed1ff16b2ab76"}
Oct 30 19:15:18.715476 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.715455    4372 scope.go:122] "RemoveContainer" containerID="9147583df277cc6d77badc7527a7cb2dff9bc6db450c17fc972ed1ff16b2ab76"
Oct 30 19:15:18.716306 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:18.716283    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w_odh-ai-gateway-infra(bc27b495-5bb4-4603-b302-14d42c209e91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w" podUID="bc27b495-5bb4-4603-b302-14d42c209e91"
Oct 30 19:15:18.717943 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.717921    4372 generic.go:356] "Generic (PLEG): container finished" podID="d1ca763d-c700-46aa-931e-2c583feaa266" containerID="dc91c224bc559dd398916ebaafdbc20f1e072566deec8b878a158ef2b164b51f" exitCode=60
Oct 30 19:15:18.718051 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.718003    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g" event={"ID":"d1ca763d-c700-46aa-931e-2c583feaa266","Type":"ContainerDied","Data":"dc91c224bc559dd398916ebaafdbc20f1e072566deec8b878a158ef2b164b51f"}
Oct 30 19:15:18.718544 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.718416    4372 scope.go:122] "RemoveContainer" containerID="dc91c224bc559dd398916ebaafdbc20f1e072566deec8b878a158ef2b164b51f"
Oct 30 19:15:18.718809 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:18.718784    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g_odh-ai-gateway-infra(d1ca763d-c700-46aa-931e-2c583feaa266)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g" podUID="d1ca763d-c700-46aa-931e-2c583feaa266"
Oct 30 19:15:18.719989 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.719973    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"
Oct 30 19:15:18.720103 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.720083    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth" event={"ID":"f546370d-4f7c-4db2-a58e-6efb9db578e3","Type":"ContainerDied","Data":"64eeefbf621d87b736f718c8a53fb58bde4583ec49e4dd214544aba8a37d038c"}
Oct 30 19:15:18.722734 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.722712    4372 generic.go:356] "Generic (PLEG): container finished" podID="fb42d228-1932-4501-8d6b-4b12c3b964e0" containerID="67e5d5227b0b48a17fde5c215c4100c076de229c6d3fc466cb4445539ae3c8de" exitCode=60
Oct 30 19:15:18.722834 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.722744    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7" event={"ID":"fb42d228-1932-4501-8d6b-4b12c3b964e0","Type":"ContainerDied","Data":"67e5d5227b0b48a17fde5c215c4100c076de229c6d3fc466cb4445539ae3c8de"}
Oct 30 19:15:18.723058 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.723039    4372 scope.go:122] "RemoveContainer" containerID="67e5d5227b0b48a17fde5c215c4100c076de229c6d3fc466cb4445539ae3c8de"
Oct 30 19:15:18.723387 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:18.723361    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7_odh-ai-gateway-infra(fb42d228-1932-4501-8d6b-4b12c3b964e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7" podUID="fb42d228-1932-4501-8d6b-4b12c3b964e0"
Oct 30 19:15:18.727241 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.727206    4372 scope.go:122] "RemoveContainer" containerID="2338bb89faf1f68cc3784d6cf6dc4d7c1a4d8d4ec469b95de51aa361435d0901"
Oct 30 19:15:18.745524 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.745499    4372 scope.go:122] "RemoveContainer" containerID="41d94aebb6eb452d054560594586fcfa097f394117fc0530f5112822857675e9"
Oct 30 19:15:18.764339 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.764313    4372 scope.go:122] "RemoveContainer" containerID="00e6a6c8359aa1e1c8cdfcb8c84c5d6379aa7a2c3395e24b74c43187bb1c99c6"
Oct 30 19:15:18.782265 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.781779    4372 scope.go:122] "RemoveContainer" containerID="fd5c1dd38bcd5c148cdedb0de95a6d825ce90abd7c96850926356ddb120bb146"
Oct 30 19:15:18.922092 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.921224    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"]
Oct 30 19:15:18.925094 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.925064    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889795-4vgth"]
Oct 30 19:15:18.984933 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:18.984886    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"
Oct 30 19:15:19.123627 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.123596    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79318342-730a-44bc-9bb4-b1e4455ebc16-kube-api-access-4hlzz\" (UniqueName: \"kubernetes.io/projected/79318342-730a-44bc-9bb4-b1e4455ebc16-kube-api-access-4hlzz\") pod \"79318342-730a-44bc-9bb4-b1e4455ebc16\" (UID: \"79318342-730a-44bc-9bb4-b1e4455ebc16\") "
Oct 30 19:15:19.123930 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.123915    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79318342-730a-44bc-9bb4-b1e4455ebc16-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79318342-730a-44bc-9bb4-b1e4455ebc16-maas-api-service-ca\") pod \"79318342-730a-44bc-9bb4-b1e4455ebc16\" (UID: \"79318342-730a-44bc-9bb4-b1e4455ebc16\") "
Oct 30 19:15:19.124326 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.124303    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79318342-730a-44bc-9bb4-b1e4455ebc16-maas-api-service-ca" pod "79318342-730a-44bc-9bb4-b1e4455ebc16" (UID: "79318342-730a-44bc-9bb4-b1e4455ebc16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:19.124529 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.124515    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79318342-730a-44bc-9bb4-b1e4455ebc16-maas-api-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.126023 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.125996    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79318342-730a-44bc-9bb4-b1e4455ebc16-kube-api-access-4hlzz" pod "79318342-730a-44bc-9bb4-b1e4455ebc16" (UID: "79318342-730a-44bc-9bb4-b1e4455ebc16"). InnerVolumeSpecName "kube-api-access-4hlzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:19.225622 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.225554    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hlzz\" (UniqueName: \"kubernetes.io/projected/79318342-730a-44bc-9bb4-b1e4455ebc16-kube-api-access-4hlzz\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.258264 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.258241    4372 scope.go:122] "RemoveContainer" containerID="408637720ea69dc6c9bdf4e5129a745b687ec122ed758c88a2cb1e89ec04cc00"
Oct 30 19:15:19.731796 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.731714    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"
Oct 30 19:15:19.732664 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.731715    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph" event={"ID":"79318342-730a-44bc-9bb4-b1e4455ebc16","Type":"ContainerDied","Data":"03bbb5d3b02e56e24dd3ef0881472568696b5797e515539e2ccf98abca871458"}
Oct 30 19:15:19.732664 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.731874    4372 scope.go:122] "RemoveContainer" containerID="ec22470f7585ba96c7bafba6a5d44a7a26c0d468cd54531f38070764b01682c5"
Oct 30 19:15:19.735977 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.735952    4372 generic.go:356] "Generic (PLEG): container finished" podID="e2fc2cfe-9426-44a2-8492-9ced609b156a" containerID="09056ee34a48ee8b32082fd58105195355fe38499ba108616cfbb9fc34cb19bb" exitCode=60
Oct 30 19:15:19.736087 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.735983    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng" event={"ID":"e2fc2cfe-9426-44a2-8492-9ced609b156a","Type":"ContainerDied","Data":"09056ee34a48ee8b32082fd58105195355fe38499ba108616cfbb9fc34cb19bb"}
Oct 30 19:15:19.736579 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.736559    4372 scope.go:122] "RemoveContainer" containerID="09056ee34a48ee8b32082fd58105195355fe38499ba108616cfbb9fc34cb19bb"
Oct 30 19:15:19.736887 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:19.736862    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng_odh-ai-gateway-infra(e2fc2cfe-9426-44a2-8492-9ced609b156a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng" podUID="e2fc2cfe-9426-44a2-8492-9ced609b156a"
Oct 30 19:15:19.748380 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.748353    4372 scope.go:122] "RemoveContainer" containerID="0daabef96c7c181738ef162a587a2e4a5c4f92bb57c01a0f5391660aeafe70a5"
Oct 30 19:15:19.749962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.749939    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"]
Oct 30 19:15:19.750327 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.750306    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj" event={"ID":"c46095e1-0538-4377-adad-69535ea14d47","Type":"ContainerStarted","Data":"21cc70e1e6d5676b68bd79e213f5213fa50ac4a12c2b6c7ce300bc5e3567f08e"}
Oct 30 19:15:19.782671 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.782529    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"]
Oct 30 19:15:19.795170 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.792585    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"]
Oct 30 19:15:19.801519 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.801193    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"]
Oct 30 19:15:19.815718 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.815682    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-304f22-29889795-jxlph"]
Oct 30 19:15:19.845386 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:19.845361    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"]
Oct 30 19:15:20.095914 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.095883    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"
Oct 30 19:15:20.160319 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.160275    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d24a5b-5eae-49f0-ae25-4540bad7ade4-kube-api-access-kbsxn\" (UniqueName: \"kubernetes.io/projected/06d24a5b-5eae-49f0-ae25-4540bad7ade4-kube-api-access-kbsxn\") pod \"06d24a5b-5eae-49f0-ae25-4540bad7ade4\" (UID: \"06d24a5b-5eae-49f0-ae25-4540bad7ade4\") "
Oct 30 19:15:20.160547 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.160526    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d24a5b-5eae-49f0-ae25-4540bad7ade4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d24a5b-5eae-49f0-ae25-4540bad7ade4-maas-api-service-ca\") pod \"06d24a5b-5eae-49f0-ae25-4540bad7ade4\" (UID: \"06d24a5b-5eae-49f0-ae25-4540bad7ade4\") "
Oct 30 19:15:20.162628 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.162583    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d24a5b-5eae-49f0-ae25-4540bad7ade4-maas-api-service-ca" pod "06d24a5b-5eae-49f0-ae25-4540bad7ade4" (UID: "06d24a5b-5eae-49f0-ae25-4540bad7ade4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.172108 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.172070    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d24a5b-5eae-49f0-ae25-4540bad7ade4-kube-api-access-kbsxn" pod "06d24a5b-5eae-49f0-ae25-4540bad7ade4" (UID: "06d24a5b-5eae-49f0-ae25-4540bad7ade4"). InnerVolumeSpecName "kube-api-access-kbsxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.202317 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.202282    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"
Oct 30 19:15:20.245084 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.245053    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"
Oct 30 19:15:20.258502 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.258479    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"
Oct 30 19:15:20.268686 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.263881    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb42d228-1932-4501-8d6b-4b12c3b964e0-kube-api-access-kjhxk\" (UniqueName: \"kubernetes.io/projected/fb42d228-1932-4501-8d6b-4b12c3b964e0-kube-api-access-kjhxk\") pod \"fb42d228-1932-4501-8d6b-4b12c3b964e0\" (UID: \"fb42d228-1932-4501-8d6b-4b12c3b964e0\") "
Oct 30 19:15:20.268686 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.264253    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb42d228-1932-4501-8d6b-4b12c3b964e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb42d228-1932-4501-8d6b-4b12c3b964e0-maas-api-service-ca\") pod \"fb42d228-1932-4501-8d6b-4b12c3b964e0\" (UID: \"fb42d228-1932-4501-8d6b-4b12c3b964e0\") "
Oct 30 19:15:20.268686 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.265439    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d24a5b-5eae-49f0-ae25-4540bad7ade4-maas-api-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.268686 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.265465    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbsxn\" (UniqueName: \"kubernetes.io/projected/06d24a5b-5eae-49f0-ae25-4540bad7ade4-kube-api-access-kbsxn\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.268686 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.265998    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb42d228-1932-4501-8d6b-4b12c3b964e0-maas-api-service-ca" pod "fb42d228-1932-4501-8d6b-4b12c3b964e0" (UID: "fb42d228-1932-4501-8d6b-4b12c3b964e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.293866 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.293843    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79318342-730a-44bc-9bb4-b1e4455ebc16" path="/var/lib/kubelet/pods/79318342-730a-44bc-9bb4-b1e4455ebc16/volumes"
Oct 30 19:15:20.294167 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.293998    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb42d228-1932-4501-8d6b-4b12c3b964e0-kube-api-access-kjhxk" pod "fb42d228-1932-4501-8d6b-4b12c3b964e0" (UID: "fb42d228-1932-4501-8d6b-4b12c3b964e0"). InnerVolumeSpecName "kube-api-access-kjhxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.294772 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.294707    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f546370d-4f7c-4db2-a58e-6efb9db578e3" path="/var/lib/kubelet/pods/f546370d-4f7c-4db2-a58e-6efb9db578e3/volumes"
Oct 30 19:15:20.366252 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.366209    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ca763d-c700-46aa-931e-2c583feaa266-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ca763d-c700-46aa-931e-2c583feaa266-maas-api-service-ca\") pod \"d1ca763d-c700-46aa-931e-2c583feaa266\" (UID: \"d1ca763d-c700-46aa-931e-2c583feaa266\") "
Oct 30 19:15:20.366372 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.366283    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc27b495-5bb4-4603-b302-14d42c209e91-kube-api-access-d2vrs\" (UniqueName: \"kubernetes.io/projected/bc27b495-5bb4-4603-b302-14d42c209e91-kube-api-access-d2vrs\") pod \"bc27b495-5bb4-4603-b302-14d42c209e91\" (UID: \"bc27b495-5bb4-4603-b302-14d42c209e91\") "
Oct 30 19:15:20.366465 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.366445    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc27b495-5bb4-4603-b302-14d42c209e91-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc27b495-5bb4-4603-b302-14d42c209e91-maas-api-service-ca\") pod \"bc27b495-5bb4-4603-b302-14d42c209e91\" (UID: \"bc27b495-5bb4-4603-b302-14d42c209e91\") "
Oct 30 19:15:20.366913 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.366885    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ca763d-c700-46aa-931e-2c583feaa266-kube-api-access-8vddc\" (UniqueName: \"kubernetes.io/projected/d1ca763d-c700-46aa-931e-2c583feaa266-kube-api-access-8vddc\") pod \"d1ca763d-c700-46aa-931e-2c583feaa266\" (UID: \"d1ca763d-c700-46aa-931e-2c583feaa266\") "
Oct 30 19:15:20.367206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.367184    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ca763d-c700-46aa-931e-2c583feaa266-maas-api-service-ca" pod "d1ca763d-c700-46aa-931e-2c583feaa266" (UID: "d1ca763d-c700-46aa-931e-2c583feaa266"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.367710 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.367689    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjhxk\" (UniqueName: \"kubernetes.io/projected/fb42d228-1932-4501-8d6b-4b12c3b964e0-kube-api-access-kjhxk\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.367817 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.367716    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ca763d-c700-46aa-931e-2c583feaa266-maas-api-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.367817 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.367733    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb42d228-1932-4501-8d6b-4b12c3b964e0-maas-api-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.368438 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.368412    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc27b495-5bb4-4603-b302-14d42c209e91-maas-api-service-ca" pod "bc27b495-5bb4-4603-b302-14d42c209e91" (UID: "bc27b495-5bb4-4603-b302-14d42c209e91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.369969 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.369936    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ca763d-c700-46aa-931e-2c583feaa266-kube-api-access-8vddc" pod "d1ca763d-c700-46aa-931e-2c583feaa266" (UID: "d1ca763d-c700-46aa-931e-2c583feaa266"). InnerVolumeSpecName "kube-api-access-8vddc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.370856 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.370815    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc27b495-5bb4-4603-b302-14d42c209e91-kube-api-access-d2vrs" pod "bc27b495-5bb4-4603-b302-14d42c209e91" (UID: "bc27b495-5bb4-4603-b302-14d42c209e91"). InnerVolumeSpecName "kube-api-access-d2vrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.469815 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.469788    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vddc\" (UniqueName: \"kubernetes.io/projected/d1ca763d-c700-46aa-931e-2c583feaa266-kube-api-access-8vddc\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.469815 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.469814    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2vrs\" (UniqueName: \"kubernetes.io/projected/bc27b495-5bb4-4603-b302-14d42c209e91-kube-api-access-d2vrs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.469959 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.469825    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc27b495-5bb4-4603-b302-14d42c209e91-maas-api-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.767850 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.767803    4372 generic.go:356] "Generic (PLEG): container finished" podID="c46095e1-0538-4377-adad-69535ea14d47" containerID="21cc70e1e6d5676b68bd79e213f5213fa50ac4a12c2b6c7ce300bc5e3567f08e" exitCode=60
Oct 30 19:15:20.768348 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.767891    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj" event={"ID":"c46095e1-0538-4377-adad-69535ea14d47","Type":"ContainerDied","Data":"21cc70e1e6d5676b68bd79e213f5213fa50ac4a12c2b6c7ce300bc5e3567f08e"}
Oct 30 19:15:20.768348 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.767990    4372 scope.go:122] "RemoveContainer" containerID="408637720ea69dc6c9bdf4e5129a745b687ec122ed758c88a2cb1e89ec04cc00"
Oct 30 19:15:20.768492 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.768456    4372 scope.go:122] "RemoveContainer" containerID="21cc70e1e6d5676b68bd79e213f5213fa50ac4a12c2b6c7ce300bc5e3567f08e"
Oct 30 19:15:20.768972 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:20.768870    4372 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj_odh-ai-gateway-infra(c46095e1-0538-4377-adad-69535ea14d47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj" podUID="c46095e1-0538-4377-adad-69535ea14d47"
Oct 30 19:15:20.770177 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.770098    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7" event={"ID":"fb42d228-1932-4501-8d6b-4b12c3b964e0","Type":"ContainerDied","Data":"09505ac8e1dfbc0f336c8bb89842f6359efcc32e708d35fb6895424eeb04c0aa"}
Oct 30 19:15:20.770549 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.770513    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"
Oct 30 19:15:20.774852 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.774824    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"]
Oct 30 19:15:20.775409 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.775326    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"
Oct 30 19:15:20.775550 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.775326    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf" event={"ID":"06d24a5b-5eae-49f0-ae25-4540bad7ade4","Type":"ContainerDied","Data":"d71475e4c92fa1e75a7022b1f5ac2a03f36ea2e6272f50c89c5ffada51876e22"}
Oct 30 19:15:20.783338 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.783310    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w" event={"ID":"bc27b495-5bb4-4603-b302-14d42c209e91","Type":"ContainerDied","Data":"950bed5fa1be9b6276ecc12389856b4cc9349e4aad6ed0f618bb6d8e8aa98d2d"}
Oct 30 19:15:20.783428 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.783409    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"
Oct 30 19:15:20.792011 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.791984    4372 scope.go:122] "RemoveContainer" containerID="67e5d5227b0b48a17fde5c215c4100c076de229c6d3fc466cb4445539ae3c8de"
Oct 30 19:15:20.793584 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.793554    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"
Oct 30 19:15:20.793669 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.793589    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g" event={"ID":"d1ca763d-c700-46aa-931e-2c583feaa266","Type":"ContainerDied","Data":"681a7a290f4332e5fb1c030d267dedd27e859a03144292a55d7344925702361b"}
Oct 30 19:15:20.810635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.810540    4372 scope.go:122] "RemoveContainer" containerID="ee2b63852ce8c5b9f2c97f539d769a654a405d761768bca259a095f21ac5aa48"
Oct 30 19:15:20.823508 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.823370    4372 scope.go:122] "RemoveContainer" containerID="9147583df277cc6d77badc7527a7cb2dff9bc6db450c17fc972ed1ff16b2ab76"
Oct 30 19:15:20.826603 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.826582    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"]
Oct 30 19:15:20.881999 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.881968    4372 scope.go:122] "RemoveContainer" containerID="dc91c224bc559dd398916ebaafdbc20f1e072566deec8b878a158ef2b164b51f"
Oct 30 19:15:20.934960 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.915042    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-srkjf"]
Oct 30 19:15:20.934960 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.921930    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"]
Oct 30 19:15:20.961875 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.961841    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"]
Oct 30 19:15:20.967234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.967212    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-21d896-29889795-5stq7"]
Oct 30 19:15:20.971685 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.971662    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"]
Oct 30 19:15:20.978723 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:20.978698    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e77b1d-29889795-qkp9w"]
Oct 30 19:15:21.006685 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:21.006504    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"]
Oct 30 19:15:21.012534 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:21.012508    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-aa6736-29889795-5b84g"]
Oct 30 19:15:21.087008 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:21.086989    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"
Oct 30 19:15:21.191908 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:21.191879    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2fc2cfe-9426-44a2-8492-9ced609b156a-kube-api-access-bbt4p\" (UniqueName: \"kubernetes.io/projected/e2fc2cfe-9426-44a2-8492-9ced609b156a-kube-api-access-bbt4p\") pod \"e2fc2cfe-9426-44a2-8492-9ced609b156a\" (UID: \"e2fc2cfe-9426-44a2-8492-9ced609b156a\") "
Oct 30 19:15:21.192091 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:21.192022    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2fc2cfe-9426-44a2-8492-9ced609b156a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2fc2cfe-9426-44a2-8492-9ced609b156a-maas-api-service-ca\") pod \"e2fc2cfe-9426-44a2-8492-9ced609b156a\" (UID: \"e2fc2cfe-9426-44a2-8492-9ced609b156a\") "
Oct 30 19:15:21.192405 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:21.192383    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2fc2cfe-9426-44a2-8492-9ced609b156a-maas-api-service-ca" pod "e2fc2cfe-9426-44a2-8492-9ced609b156a" (UID: "e2fc2cfe-9426-44a2-8492-9ced609b156a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:21.192685 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:21.192669    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2fc2cfe-9426-44a2-8492-9ced609b156a-maas-api-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.194245 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:21.194223    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fc2cfe-9426-44a2-8492-9ced609b156a-kube-api-access-bbt4p" pod "e2fc2cfe-9426-44a2-8492-9ced609b156a" (UID: "e2fc2cfe-9426-44a2-8492-9ced609b156a"). InnerVolumeSpecName "kube-api-access-bbt4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:21.293874 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:21.293842    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbt4p\" (UniqueName: \"kubernetes.io/projected/e2fc2cfe-9426-44a2-8492-9ced609b156a-kube-api-access-bbt4p\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.809008 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:21.808982    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"
Oct 30 19:15:21.809468 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:21.808981    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng" event={"ID":"e2fc2cfe-9426-44a2-8492-9ced609b156a","Type":"ContainerDied","Data":"e51ad53f317e86c1479d1f785ee2db7a8057eff84b20fc79144b3d5a0dec11fa"}
Oct 30 19:15:21.809468 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:21.809241    4372 scope.go:122] "RemoveContainer" containerID="09056ee34a48ee8b32082fd58105195355fe38499ba108616cfbb9fc34cb19bb"
Oct 30 19:15:21.960411 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:21.960367    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"]
Oct 30 19:15:21.965552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:21.965528    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8a92cc-29889795-qhkng"]
Oct 30 19:15:22.121895 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.121869    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"
Oct 30 19:15:22.158515 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.158489    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46095e1-0538-4377-adad-69535ea14d47-kube-api-access-fpvhp\" (UniqueName: \"kubernetes.io/projected/c46095e1-0538-4377-adad-69535ea14d47-kube-api-access-fpvhp\") pod \"c46095e1-0538-4377-adad-69535ea14d47\" (UID: \"c46095e1-0538-4377-adad-69535ea14d47\") "
Oct 30 19:15:22.158644 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.158586    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c46095e1-0538-4377-adad-69535ea14d47-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c46095e1-0538-4377-adad-69535ea14d47-maas-api-service-ca\") pod \"c46095e1-0538-4377-adad-69535ea14d47\" (UID: \"c46095e1-0538-4377-adad-69535ea14d47\") "
Oct 30 19:15:22.158934 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.158914    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c46095e1-0538-4377-adad-69535ea14d47-maas-api-service-ca" pod "c46095e1-0538-4377-adad-69535ea14d47" (UID: "c46095e1-0538-4377-adad-69535ea14d47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:22.159221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.159206    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c46095e1-0538-4377-adad-69535ea14d47-maas-api-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.160942 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.160900    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46095e1-0538-4377-adad-69535ea14d47-kube-api-access-fpvhp" pod "c46095e1-0538-4377-adad-69535ea14d47" (UID: "c46095e1-0538-4377-adad-69535ea14d47"). InnerVolumeSpecName "kube-api-access-fpvhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:22.260371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.260342    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpvhp\" (UniqueName: \"kubernetes.io/projected/c46095e1-0538-4377-adad-69535ea14d47-kube-api-access-fpvhp\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.268093 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.268067    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d24a5b-5eae-49f0-ae25-4540bad7ade4" path="/var/lib/kubelet/pods/06d24a5b-5eae-49f0-ae25-4540bad7ade4/volumes"
Oct 30 19:15:22.268593 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.268577    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc27b495-5bb4-4603-b302-14d42c209e91" path="/var/lib/kubelet/pods/bc27b495-5bb4-4603-b302-14d42c209e91/volumes"
Oct 30 19:15:22.269030 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.269015    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ca763d-c700-46aa-931e-2c583feaa266" path="/var/lib/kubelet/pods/d1ca763d-c700-46aa-931e-2c583feaa266/volumes"
Oct 30 19:15:22.269953 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.269935    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2fc2cfe-9426-44a2-8492-9ced609b156a" path="/var/lib/kubelet/pods/e2fc2cfe-9426-44a2-8492-9ced609b156a/volumes"
Oct 30 19:15:22.270454 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.270436    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb42d228-1932-4501-8d6b-4b12c3b964e0" path="/var/lib/kubelet/pods/fb42d228-1932-4501-8d6b-4b12c3b964e0/volumes"
Oct 30 19:15:22.400651 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.400586    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:15:22.826753 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.826725    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"
Oct 30 19:15:22.827278 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.826789    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj" event={"ID":"c46095e1-0538-4377-adad-69535ea14d47","Type":"ContainerDied","Data":"c2e8cd4dd836b070adf46fd357d460096aa758ed8ac800286a7bf5cc3b3c1513"}
Oct 30 19:15:22.827278 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.826866    4372 scope.go:122] "RemoveContainer" containerID="21cc70e1e6d5676b68bd79e213f5213fa50ac4a12c2b6c7ce300bc5e3567f08e"
Oct 30 19:15:22.867699 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.867673    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"]
Oct 30 19:15:22.883533 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:22.883512    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889795-nrltj"]
Oct 30 19:15:24.267698 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:24.267666    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c46095e1-0538-4377-adad-69535ea14d47" path="/var/lib/kubelet/pods/c46095e1-0538-4377-adad-69535ea14d47/volumes"
Oct 30 19:15:27.697351 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:27.697314    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:15:57.000219 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:15:57.000112    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:15:57.698652 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:15:57.698615    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:16:02.556214 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.556173    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"]
Oct 30 19:16:02.557197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557174    4372 cpu_manager.go:414] "containerMap: removing container" podUID="bc27b495-5bb4-4603-b302-14d42c209e91" containerName="cleanup"
Oct 30 19:16:02.557197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557198    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc27b495-5bb4-4603-b302-14d42c209e91" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557210    4372 cpu_manager.go:414] "containerMap: removing container" podUID="e2fc2cfe-9426-44a2-8492-9ced609b156a" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557218    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2fc2cfe-9426-44a2-8492-9ced609b156a" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557228    4372 cpu_manager.go:414] "containerMap: removing container" podUID="c46095e1-0538-4377-adad-69535ea14d47" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557235    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46095e1-0538-4377-adad-69535ea14d47" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557252    4372 cpu_manager.go:414] "containerMap: removing container" podUID="79318342-730a-44bc-9bb4-b1e4455ebc16" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557258    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79318342-730a-44bc-9bb4-b1e4455ebc16" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557268    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d1ca763d-c700-46aa-931e-2c583feaa266" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557275    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ca763d-c700-46aa-931e-2c583feaa266" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557289    4372 cpu_manager.go:414] "containerMap: removing container" podUID="c46095e1-0538-4377-adad-69535ea14d47" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557295    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46095e1-0538-4377-adad-69535ea14d47" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557308    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f546370d-4f7c-4db2-a58e-6efb9db578e3" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557315    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f546370d-4f7c-4db2-a58e-6efb9db578e3" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557331    4372 cpu_manager.go:414] "containerMap: removing container" podUID="bc27b495-5bb4-4603-b302-14d42c209e91" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557338    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc27b495-5bb4-4603-b302-14d42c209e91" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557351    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f546370d-4f7c-4db2-a58e-6efb9db578e3" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557358    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f546370d-4f7c-4db2-a58e-6efb9db578e3" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557378    4372 cpu_manager.go:414] "containerMap: removing container" podUID="06d24a5b-5eae-49f0-ae25-4540bad7ade4" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557386    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d24a5b-5eae-49f0-ae25-4540bad7ade4" containerName="cleanup"
Oct 30 19:16:02.557390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557397    4372 cpu_manager.go:414] "containerMap: removing container" podUID="e2fc2cfe-9426-44a2-8492-9ced609b156a" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557403    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2fc2cfe-9426-44a2-8492-9ced609b156a" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557419    4372 cpu_manager.go:414] "containerMap: removing container" podUID="06d24a5b-5eae-49f0-ae25-4540bad7ade4" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557426    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d24a5b-5eae-49f0-ae25-4540bad7ade4" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557449    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d1ca763d-c700-46aa-931e-2c583feaa266" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557456    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ca763d-c700-46aa-931e-2c583feaa266" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557476    4372 cpu_manager.go:414] "containerMap: removing container" podUID="79318342-730a-44bc-9bb4-b1e4455ebc16" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557484    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79318342-730a-44bc-9bb4-b1e4455ebc16" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557516    4372 cpu_manager.go:414] "containerMap: removing container" podUID="fb42d228-1932-4501-8d6b-4b12c3b964e0" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557525    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb42d228-1932-4501-8d6b-4b12c3b964e0" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557538    4372 cpu_manager.go:414] "containerMap: removing container" podUID="06d24a5b-5eae-49f0-ae25-4540bad7ade4" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557546    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d24a5b-5eae-49f0-ae25-4540bad7ade4" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557560    4372 cpu_manager.go:414] "containerMap: removing container" podUID="bc27b495-5bb4-4603-b302-14d42c209e91" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557566    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc27b495-5bb4-4603-b302-14d42c209e91" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557578    4372 cpu_manager.go:414] "containerMap: removing container" podUID="fb42d228-1932-4501-8d6b-4b12c3b964e0" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557585    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb42d228-1932-4501-8d6b-4b12c3b964e0" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557784    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46095e1-0538-4377-adad-69535ea14d47" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557798    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="f546370d-4f7c-4db2-a58e-6efb9db578e3" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557810    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc27b495-5bb4-4603-b302-14d42c209e91" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557819    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="f546370d-4f7c-4db2-a58e-6efb9db578e3" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557828    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2fc2cfe-9426-44a2-8492-9ced609b156a" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557839    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="79318342-730a-44bc-9bb4-b1e4455ebc16" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557848    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d24a5b-5eae-49f0-ae25-4540bad7ade4" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557856    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2fc2cfe-9426-44a2-8492-9ced609b156a" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557866    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d24a5b-5eae-49f0-ae25-4540bad7ade4" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557876    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ca763d-c700-46aa-931e-2c583feaa266" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557885    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46095e1-0538-4377-adad-69535ea14d47" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557900    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb42d228-1932-4501-8d6b-4b12c3b964e0" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557910    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc27b495-5bb4-4603-b302-14d42c209e91" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557918    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb42d228-1932-4501-8d6b-4b12c3b964e0" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557929    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb42d228-1932-4501-8d6b-4b12c3b964e0" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557961    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc27b495-5bb4-4603-b302-14d42c209e91" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557971    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46095e1-0538-4377-adad-69535ea14d47" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557980    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="79318342-730a-44bc-9bb4-b1e4455ebc16" containerName="cleanup"
Oct 30 19:16:02.558434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.557990    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ca763d-c700-46aa-931e-2c583feaa266" containerName="cleanup"
Oct 30 19:16:02.562515 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.562489    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:02.571004 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.570959    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"agc-cred-b19eaa306199\""
Oct 30 19:16:02.571004 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.570959    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"openshift-service-ca.crt\""
Oct 30 19:16:02.572342 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.572322    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"kube-root-ca.crt\""
Oct 30 19:16:02.573556 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.572527    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"payload-processing-external-model-e2e--b99c0f48-dockercfg-dp5v2\""
Oct 30 19:16:02.573556 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.572570    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d72beb\""
Oct 30 19:16:02.573556 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.572663    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"routing-overlay\""
Oct 30 19:16:02.577014 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.576989    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"]
Oct 30 19:16:02.872150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.871845    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cb229c9-ca25-4734-afdc-6001d9083817-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:02.872150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.871948    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cb229c9-ca25-4734-afdc-6001d9083817-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:02.872150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.872070    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb229c9-ca25-4734-afdc-6001d9083817-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:02.872471 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.872180    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2dsf\" (UniqueName: \"kubernetes.io/projected/3cb229c9-ca25-4734-afdc-6001d9083817-kube-api-access-l2dsf\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:02.974586 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.974540    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cb229c9-ca25-4734-afdc-6001d9083817-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:02.975538 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.974965    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb229c9-ca25-4734-afdc-6001d9083817-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:02.978953 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.978925    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2dsf\" (UniqueName: \"kubernetes.io/projected/3cb229c9-ca25-4734-afdc-6001d9083817-kube-api-access-l2dsf\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:02.979375 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.979352    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cb229c9-ca25-4734-afdc-6001d9083817-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:02.998707 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.983769    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb229c9-ca25-4734-afdc-6001d9083817-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:03.000321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:02.986096    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cb229c9-ca25-4734-afdc-6001d9083817-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:03.016261 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:03.015996    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cb229c9-ca25-4734-afdc-6001d9083817-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:03.026144 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:03.021246    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2dsf\" (UniqueName: \"kubernetes.io/projected/3cb229c9-ca25-4734-afdc-6001d9083817-kube-api-access-l2dsf\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:03.181633 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:03.181567    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:03.531293 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:03.531262    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"]
Oct 30 19:16:03.532730 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:16:03.532689    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb229c9_ca25_4734_afdc_6001d9083817.slice/crio-de1b3c7102fae4de895add81ea918e2aa5fdfd1f18c9dcafda34b89a7127f273 WatchSource:0}: Error finding container de1b3c7102fae4de895add81ea918e2aa5fdfd1f18c9dcafda34b89a7127f273: Status 404 returned error can't find the container with id de1b3c7102fae4de895add81ea918e2aa5fdfd1f18c9dcafda34b89a7127f273
Oct 30 19:16:04.244180 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:04.244112    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m" event={"ID":"3cb229c9-ca25-4734-afdc-6001d9083817","Type":"ContainerStarted","Data":"28b2ad8e1778d3811937701db710ad4f0598a409128276fc5380ac226a206287"}
Oct 30 19:16:04.244583 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:04.244207    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:04.244583 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:04.244227    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m" event={"ID":"3cb229c9-ca25-4734-afdc-6001d9083817","Type":"ContainerStarted","Data":"de1b3c7102fae4de895add81ea918e2aa5fdfd1f18c9dcafda34b89a7127f273"}
Oct 30 19:16:04.307665 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:04.307597    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m" podStartSLOduration=2.307576548 podStartE2EDuration="2.307576548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:04.295454823 +0000 UTC m=+2578.163873196" watchObservedRunningTime="2026-10-30 19:16:04.307576548 +0000 UTC m=+2578.175994879"
Oct 30 19:16:15.258487 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:15.258457    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:16:27.697332 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:16:27.697288    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:16:44.966505 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:16:44.966473    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:16:57.696482 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:16:57.696446    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:17:08.150427 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:08.150392    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"]
Oct 30 19:17:08.150880 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:08.150579    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw" podUID="4bf56cfc-1bc7-4221-a324-f6b885c80415" containerName="istio-proxy" containerID="cri-o://716dd14a1fd586c8d7a019b0c2d3e3343a724e986db15605f8edb15066bfc54a" gracePeriod=30
Oct 30 19:17:13.544476 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.544448    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:17:13.651146 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.651098    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bf56cfc-1bc7-4221-a324-f6b885c80415-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf56cfc-1bc7-4221-a324-f6b885c80415-istiod-ca-cert\") pod \"4bf56cfc-1bc7-4221-a324-f6b885c80415\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") "
Oct 30 19:17:13.651346 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.651165    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-podinfo\") pod \"4bf56cfc-1bc7-4221-a324-f6b885c80415\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") "
Oct 30 19:17:13.651346 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.651194    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-workload-socket\") pod \"4bf56cfc-1bc7-4221-a324-f6b885c80415\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") "
Oct 30 19:17:13.651346 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.651273    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf56cfc-1bc7-4221-a324-f6b885c80415-kube-api-access-hvp4w\" (UniqueName: \"kubernetes.io/projected/4bf56cfc-1bc7-4221-a324-f6b885c80415-kube-api-access-hvp4w\") pod \"4bf56cfc-1bc7-4221-a324-f6b885c80415\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") "
Oct 30 19:17:13.651346 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.651298    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-data\") pod \"4bf56cfc-1bc7-4221-a324-f6b885c80415\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") "
Oct 30 19:17:13.651346 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.651317    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-workload-certs\") pod \"4bf56cfc-1bc7-4221-a324-f6b885c80415\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") "
Oct 30 19:17:13.651665 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.651369    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-credential-socket\") pod \"4bf56cfc-1bc7-4221-a324-f6b885c80415\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") "
Oct 30 19:17:13.651665 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.651420    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-envoy\") pod \"4bf56cfc-1bc7-4221-a324-f6b885c80415\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") "
Oct 30 19:17:13.651665 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.651465    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-token\" (UniqueName: \"kubernetes.io/projected/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-token\") pod \"4bf56cfc-1bc7-4221-a324-f6b885c80415\" (UID: \"4bf56cfc-1bc7-4221-a324-f6b885c80415\") "
Oct 30 19:17:13.651665 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.651500    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf56cfc-1bc7-4221-a324-f6b885c80415-istiod-ca-cert" pod "4bf56cfc-1bc7-4221-a324-f6b885c80415" (UID: "4bf56cfc-1bc7-4221-a324-f6b885c80415"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:13.651889 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.651831    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-data" pod "4bf56cfc-1bc7-4221-a324-f6b885c80415" (UID: "4bf56cfc-1bc7-4221-a324-f6b885c80415"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:13.651889 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.651867    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf56cfc-1bc7-4221-a324-f6b885c80415-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.652070 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.652050    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-workload-certs" pod "4bf56cfc-1bc7-4221-a324-f6b885c80415" (UID: "4bf56cfc-1bc7-4221-a324-f6b885c80415"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:13.652251 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.652230    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-credential-socket" pod "4bf56cfc-1bc7-4221-a324-f6b885c80415" (UID: "4bf56cfc-1bc7-4221-a324-f6b885c80415"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:13.652409 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.652387    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-workload-socket" pod "4bf56cfc-1bc7-4221-a324-f6b885c80415" (UID: "4bf56cfc-1bc7-4221-a324-f6b885c80415"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:13.655013 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.654826    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-podinfo" pod "4bf56cfc-1bc7-4221-a324-f6b885c80415" (UID: "4bf56cfc-1bc7-4221-a324-f6b885c80415"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:13.655956 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.655933    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-envoy" pod "4bf56cfc-1bc7-4221-a324-f6b885c80415" (UID: "4bf56cfc-1bc7-4221-a324-f6b885c80415"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:13.656077 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.655937    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-token" pod "4bf56cfc-1bc7-4221-a324-f6b885c80415" (UID: "4bf56cfc-1bc7-4221-a324-f6b885c80415"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:13.656077 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.655985    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf56cfc-1bc7-4221-a324-f6b885c80415-kube-api-access-hvp4w" pod "4bf56cfc-1bc7-4221-a324-f6b885c80415" (UID: "4bf56cfc-1bc7-4221-a324-f6b885c80415"). InnerVolumeSpecName "kube-api-access-hvp4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:13.753917 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.753879    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.753917 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.753913    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.753917 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.753923    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.754234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.753933    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvp4w\" (UniqueName: \"kubernetes.io/projected/4bf56cfc-1bc7-4221-a324-f6b885c80415-kube-api-access-hvp4w\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.754234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.753944    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.754234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.753952    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.754234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.753960    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:13.754234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:13.753968    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf56cfc-1bc7-4221-a324-f6b885c80415-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.150010 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:14.149973    4372 generic.go:356] "Generic (PLEG): container finished" podID="4bf56cfc-1bc7-4221-a324-f6b885c80415" containerID="716dd14a1fd586c8d7a019b0c2d3e3343a724e986db15605f8edb15066bfc54a" exitCode=0
Oct 30 19:17:14.150241 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:14.150069    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw" event={"ID":"4bf56cfc-1bc7-4221-a324-f6b885c80415","Type":"ContainerDied","Data":"716dd14a1fd586c8d7a019b0c2d3e3343a724e986db15605f8edb15066bfc54a"}
Oct 30 19:17:14.150241 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:14.150120    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw" event={"ID":"4bf56cfc-1bc7-4221-a324-f6b885c80415","Type":"ContainerDied","Data":"23ee48a6c923f66993d9552acce656d06606ce7f62a43e1b2d5a67b7a04a54b8"}
Oct 30 19:17:14.150241 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:14.150197    4372 scope.go:122] "RemoveContainer" containerID="716dd14a1fd586c8d7a019b0c2d3e3343a724e986db15605f8edb15066bfc54a"
Oct 30 19:17:14.150241 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:14.150204    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"
Oct 30 19:17:14.165541 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:14.165520    4372 scope.go:122] "RemoveContainer" containerID="716dd14a1fd586c8d7a019b0c2d3e3343a724e986db15605f8edb15066bfc54a"
Oct 30 19:17:14.165819 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:17:14.165799    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716dd14a1fd586c8d7a019b0c2d3e3343a724e986db15605f8edb15066bfc54a\": container with ID starting with 716dd14a1fd586c8d7a019b0c2d3e3343a724e986db15605f8edb15066bfc54a not found: ID does not exist" containerID="716dd14a1fd586c8d7a019b0c2d3e3343a724e986db15605f8edb15066bfc54a"
Oct 30 19:17:14.165891 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:14.165835    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716dd14a1fd586c8d7a019b0c2d3e3343a724e986db15605f8edb15066bfc54a"} err="failed to get container status \"716dd14a1fd586c8d7a019b0c2d3e3343a724e986db15605f8edb15066bfc54a\": rpc error: code = NotFound desc = could not find container \"716dd14a1fd586c8d7a019b0c2d3e3343a724e986db15605f8edb15066bfc54a\": container with ID starting with 716dd14a1fd586c8d7a019b0c2d3e3343a724e986db15605f8edb15066bfc54a not found: ID does not exist"
Oct 30 19:17:14.182252 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:14.182222    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"]
Oct 30 19:17:14.189813 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:14.189788    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-jrbbw"]
Oct 30 19:17:14.267664 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:14.267635    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bf56cfc-1bc7-4221-a324-f6b885c80415" path="/var/lib/kubelet/pods/4bf56cfc-1bc7-4221-a324-f6b885c80415/volumes"
Oct 30 19:17:16.328250 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:16.328206    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"]
Oct 30 19:17:21.329072 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:21.328988    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m" podUID="3cb229c9-ca25-4734-afdc-6001d9083817" containerName="payload-processing" containerID="cri-o://28b2ad8e1778d3811937701db710ad4f0598a409128276fc5380ac226a206287" gracePeriod=25
Oct 30 19:17:21.621994 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:21.621967    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:17:21.693243 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:21.693214    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb229c9-ca25-4734-afdc-6001d9083817-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cb229c9-ca25-4734-afdc-6001d9083817-provider-credentials\") pod \"3cb229c9-ca25-4734-afdc-6001d9083817\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") "
Oct 30 19:17:21.693410 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:21.693372    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb229c9-ca25-4734-afdc-6001d9083817-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb229c9-ca25-4734-afdc-6001d9083817-plugins-config-volume\") pod \"3cb229c9-ca25-4734-afdc-6001d9083817\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") "
Oct 30 19:17:21.693527 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:21.693513    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb229c9-ca25-4734-afdc-6001d9083817-kube-api-access-l2dsf\" (UniqueName: \"kubernetes.io/projected/3cb229c9-ca25-4734-afdc-6001d9083817-kube-api-access-l2dsf\") pod \"3cb229c9-ca25-4734-afdc-6001d9083817\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") "
Oct 30 19:17:21.693621 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:21.693598    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb229c9-ca25-4734-afdc-6001d9083817-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cb229c9-ca25-4734-afdc-6001d9083817-routing-overlay-volume\") pod \"3cb229c9-ca25-4734-afdc-6001d9083817\" (UID: \"3cb229c9-ca25-4734-afdc-6001d9083817\") "
Oct 30 19:17:21.694165 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:21.693823    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb229c9-ca25-4734-afdc-6001d9083817-plugins-config-volume" pod "3cb229c9-ca25-4734-afdc-6001d9083817" (UID: "3cb229c9-ca25-4734-afdc-6001d9083817"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:21.694165 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:21.694080    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb229c9-ca25-4734-afdc-6001d9083817-routing-overlay-volume" pod "3cb229c9-ca25-4734-afdc-6001d9083817" (UID: "3cb229c9-ca25-4734-afdc-6001d9083817"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:21.694393 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:21.694368    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb229c9-ca25-4734-afdc-6001d9083817-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:21.694465 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:21.694399    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cb229c9-ca25-4734-afdc-6001d9083817-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:21.696212 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:21.696188    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb229c9-ca25-4734-afdc-6001d9083817-provider-credentials" pod "3cb229c9-ca25-4734-afdc-6001d9083817" (UID: "3cb229c9-ca25-4734-afdc-6001d9083817"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:21.696432 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:21.696410    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb229c9-ca25-4734-afdc-6001d9083817-kube-api-access-l2dsf" pod "3cb229c9-ca25-4734-afdc-6001d9083817" (UID: "3cb229c9-ca25-4734-afdc-6001d9083817"). InnerVolumeSpecName "kube-api-access-l2dsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:21.816089 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:21.796808    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2dsf\" (UniqueName: \"kubernetes.io/projected/3cb229c9-ca25-4734-afdc-6001d9083817-kube-api-access-l2dsf\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:21.816089 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:21.816013    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cb229c9-ca25-4734-afdc-6001d9083817-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:22.274970 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:22.274936    4372 generic.go:356] "Generic (PLEG): container finished" podID="3cb229c9-ca25-4734-afdc-6001d9083817" containerID="28b2ad8e1778d3811937701db710ad4f0598a409128276fc5380ac226a206287" exitCode=0
Oct 30 19:17:22.275449 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:22.274980    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"
Oct 30 19:17:22.275449 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:22.275018    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m" event={"ID":"3cb229c9-ca25-4734-afdc-6001d9083817","Type":"ContainerDied","Data":"28b2ad8e1778d3811937701db710ad4f0598a409128276fc5380ac226a206287"}
Oct 30 19:17:22.275449 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:22.275071    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m" event={"ID":"3cb229c9-ca25-4734-afdc-6001d9083817","Type":"ContainerDied","Data":"de1b3c7102fae4de895add81ea918e2aa5fdfd1f18c9dcafda34b89a7127f273"}
Oct 30 19:17:22.275449 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:22.275101    4372 scope.go:122] "RemoveContainer" containerID="28b2ad8e1778d3811937701db710ad4f0598a409128276fc5380ac226a206287"
Oct 30 19:17:22.286573 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:22.286554    4372 scope.go:122] "RemoveContainer" containerID="28b2ad8e1778d3811937701db710ad4f0598a409128276fc5380ac226a206287"
Oct 30 19:17:22.286849 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:17:22.286832    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28b2ad8e1778d3811937701db710ad4f0598a409128276fc5380ac226a206287\": container with ID starting with 28b2ad8e1778d3811937701db710ad4f0598a409128276fc5380ac226a206287 not found: ID does not exist" containerID="28b2ad8e1778d3811937701db710ad4f0598a409128276fc5380ac226a206287"
Oct 30 19:17:22.286905 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:22.286857    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b2ad8e1778d3811937701db710ad4f0598a409128276fc5380ac226a206287"} err="failed to get container status \"28b2ad8e1778d3811937701db710ad4f0598a409128276fc5380ac226a206287\": rpc error: code = NotFound desc = could not find container \"28b2ad8e1778d3811937701db710ad4f0598a409128276fc5380ac226a206287\": container with ID starting with 28b2ad8e1778d3811937701db710ad4f0598a409128276fc5380ac226a206287 not found: ID does not exist"
Oct 30 19:17:22.298977 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:22.298955    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"]
Oct 30 19:17:22.303719 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:22.303700    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-69946ll9m"]
Oct 30 19:17:24.272187 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:24.272154    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb229c9-ca25-4734-afdc-6001d9083817" path="/var/lib/kubelet/pods/3cb229c9-ca25-4734-afdc-6001d9083817/volumes"
Oct 30 19:17:27.696448 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:17:27.696410    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:17:28.143908 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:28.143870    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"]
Oct 30 19:17:32.469305 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:32.469271    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:17:33.144195 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.144152    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7" podUID="6882bbe0-5cf3-4d63-868e-a9588b94bced" containerName="payload-processing" containerID="cri-o://a2b35bbc6f1a378d2a5a9d0c42392d9a58512b4b0b9d036995036929120d4cd2" gracePeriod=25
Oct 30 19:17:33.502200 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.497603    4372 generic.go:356] "Generic (PLEG): container finished" podID="6882bbe0-5cf3-4d63-868e-a9588b94bced" containerID="a2b35bbc6f1a378d2a5a9d0c42392d9a58512b4b0b9d036995036929120d4cd2" exitCode=0
Oct 30 19:17:33.502200 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.497814    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7" event={"ID":"6882bbe0-5cf3-4d63-868e-a9588b94bced","Type":"ContainerDied","Data":"a2b35bbc6f1a378d2a5a9d0c42392d9a58512b4b0b9d036995036929120d4cd2"}
Oct 30 19:17:33.693181 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.693154    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:17:33.711308 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.711275    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6882bbe0-5cf3-4d63-868e-a9588b94bced-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6882bbe0-5cf3-4d63-868e-a9588b94bced-routing-overlay-volume\") pod \"6882bbe0-5cf3-4d63-868e-a9588b94bced\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") "
Oct 30 19:17:33.711465 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.711446    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6882bbe0-5cf3-4d63-868e-a9588b94bced-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6882bbe0-5cf3-4d63-868e-a9588b94bced-provider-credentials\") pod \"6882bbe0-5cf3-4d63-868e-a9588b94bced\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") "
Oct 30 19:17:33.711615 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.711599    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6882bbe0-5cf3-4d63-868e-a9588b94bced-kube-api-access-64cgb\" (UniqueName: \"kubernetes.io/projected/6882bbe0-5cf3-4d63-868e-a9588b94bced-kube-api-access-64cgb\") pod \"6882bbe0-5cf3-4d63-868e-a9588b94bced\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") "
Oct 30 19:17:33.711682 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.711662    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6882bbe0-5cf3-4d63-868e-a9588b94bced-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6882bbe0-5cf3-4d63-868e-a9588b94bced-plugins-config-volume\") pod \"6882bbe0-5cf3-4d63-868e-a9588b94bced\" (UID: \"6882bbe0-5cf3-4d63-868e-a9588b94bced\") "
Oct 30 19:17:33.711742 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.711682    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6882bbe0-5cf3-4d63-868e-a9588b94bced-routing-overlay-volume" pod "6882bbe0-5cf3-4d63-868e-a9588b94bced" (UID: "6882bbe0-5cf3-4d63-868e-a9588b94bced"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:33.712254 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.712223    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6882bbe0-5cf3-4d63-868e-a9588b94bced-plugins-config-volume" pod "6882bbe0-5cf3-4d63-868e-a9588b94bced" (UID: "6882bbe0-5cf3-4d63-868e-a9588b94bced"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:33.712489 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.712460    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6882bbe0-5cf3-4d63-868e-a9588b94bced-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.712625 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.712492    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6882bbe0-5cf3-4d63-868e-a9588b94bced-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.729674 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.715010    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6882bbe0-5cf3-4d63-868e-a9588b94bced-provider-credentials" pod "6882bbe0-5cf3-4d63-868e-a9588b94bced" (UID: "6882bbe0-5cf3-4d63-868e-a9588b94bced"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:33.729674 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.715608    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6882bbe0-5cf3-4d63-868e-a9588b94bced-kube-api-access-64cgb" pod "6882bbe0-5cf3-4d63-868e-a9588b94bced" (UID: "6882bbe0-5cf3-4d63-868e-a9588b94bced"). InnerVolumeSpecName "kube-api-access-64cgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:33.824350 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.824316    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6882bbe0-5cf3-4d63-868e-a9588b94bced-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.824508 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:33.824360    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64cgb\" (UniqueName: \"kubernetes.io/projected/6882bbe0-5cf3-4d63-868e-a9588b94bced-kube-api-access-64cgb\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.506856 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:34.506819    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"
Oct 30 19:17:34.507326 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:34.506864    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7" event={"ID":"6882bbe0-5cf3-4d63-868e-a9588b94bced","Type":"ContainerDied","Data":"9ac339e64c926de478a247e7c1e9eb67a5db2f6149951a6e67765ec4b3e4dfcc"}
Oct 30 19:17:34.507326 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:34.506929    4372 scope.go:122] "RemoveContainer" containerID="a2b35bbc6f1a378d2a5a9d0c42392d9a58512b4b0b9d036995036929120d4cd2"
Oct 30 19:17:34.530255 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:34.530204    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"]
Oct 30 19:17:34.535551 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:34.535529    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-79c559hz7"]
Oct 30 19:17:36.268021 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:36.267985    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6882bbe0-5cf3-4d63-868e-a9588b94bced" path="/var/lib/kubelet/pods/6882bbe0-5cf3-4d63-868e-a9588b94bced/volumes"
Oct 30 19:17:37.992605 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.992572    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"]
Oct 30 19:17:37.993359 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993342    4372 cpu_manager.go:414] "containerMap: removing container" podUID="fb42d228-1932-4501-8d6b-4b12c3b964e0" containerName="cleanup"
Oct 30 19:17:37.993359 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993360    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb42d228-1932-4501-8d6b-4b12c3b964e0" containerName="cleanup"
Oct 30 19:17:37.993463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993380    4372 cpu_manager.go:414] "containerMap: removing container" podUID="4bf56cfc-1bc7-4221-a324-f6b885c80415" containerName="istio-proxy"
Oct 30 19:17:37.993463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993385    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf56cfc-1bc7-4221-a324-f6b885c80415" containerName="istio-proxy"
Oct 30 19:17:37.993463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993406    4372 cpu_manager.go:414] "containerMap: removing container" podUID="c46095e1-0538-4377-adad-69535ea14d47" containerName="cleanup"
Oct 30 19:17:37.993463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993412    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46095e1-0538-4377-adad-69535ea14d47" containerName="cleanup"
Oct 30 19:17:37.993463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993427    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f546370d-4f7c-4db2-a58e-6efb9db578e3" containerName="cleanup"
Oct 30 19:17:37.993463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993431    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f546370d-4f7c-4db2-a58e-6efb9db578e3" containerName="cleanup"
Oct 30 19:17:37.993463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993439    4372 cpu_manager.go:414] "containerMap: removing container" podUID="6882bbe0-5cf3-4d63-868e-a9588b94bced" containerName="payload-processing"
Oct 30 19:17:37.993463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993444    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6882bbe0-5cf3-4d63-868e-a9588b94bced" containerName="payload-processing"
Oct 30 19:17:37.993463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993455    4372 cpu_manager.go:414] "containerMap: removing container" podUID="e2fc2cfe-9426-44a2-8492-9ced609b156a" containerName="cleanup"
Oct 30 19:17:37.993463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993459    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2fc2cfe-9426-44a2-8492-9ced609b156a" containerName="cleanup"
Oct 30 19:17:37.993463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993467    4372 cpu_manager.go:414] "containerMap: removing container" podUID="3cb229c9-ca25-4734-afdc-6001d9083817" containerName="payload-processing"
Oct 30 19:17:37.993825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993472    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb229c9-ca25-4734-afdc-6001d9083817" containerName="payload-processing"
Oct 30 19:17:37.993825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993485    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d1ca763d-c700-46aa-931e-2c583feaa266" containerName="cleanup"
Oct 30 19:17:37.993825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993489    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ca763d-c700-46aa-931e-2c583feaa266" containerName="cleanup"
Oct 30 19:17:37.993825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993502    4372 cpu_manager.go:414] "containerMap: removing container" podUID="79318342-730a-44bc-9bb4-b1e4455ebc16" containerName="cleanup"
Oct 30 19:17:37.993825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993509    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79318342-730a-44bc-9bb4-b1e4455ebc16" containerName="cleanup"
Oct 30 19:17:37.993825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993639    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf56cfc-1bc7-4221-a324-f6b885c80415" containerName="istio-proxy"
Oct 30 19:17:37.993825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993650    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="f546370d-4f7c-4db2-a58e-6efb9db578e3" containerName="cleanup"
Oct 30 19:17:37.993825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993655    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="6882bbe0-5cf3-4d63-868e-a9588b94bced" containerName="payload-processing"
Oct 30 19:17:37.993825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993662    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2fc2cfe-9426-44a2-8492-9ced609b156a" containerName="cleanup"
Oct 30 19:17:37.993825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993668    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="79318342-730a-44bc-9bb4-b1e4455ebc16" containerName="cleanup"
Oct 30 19:17:37.993825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993675    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d24a5b-5eae-49f0-ae25-4540bad7ade4" containerName="cleanup"
Oct 30 19:17:37.993825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993683    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb229c9-ca25-4734-afdc-6001d9083817" containerName="payload-processing"
Oct 30 19:17:37.993825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.993689    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ca763d-c700-46aa-931e-2c583feaa266" containerName="cleanup"
Oct 30 19:17:37.998777 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:37.998761    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:38.008482 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.008460    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"kube-root-ca.crt\""
Oct 30 19:17:38.008635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.008494    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"agc-cred-b77e998367e9\""
Oct 30 19:17:38.008635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.008510    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"routing-overlay\""
Oct 30 19:17:38.009172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.009150    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"agc-cred-56125a628169\""
Oct 30 19:17:38.009958 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.009940    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"openshift-service-ca.crt\""
Oct 30 19:17:38.010087 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.010057    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"]
Oct 30 19:17:38.010205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.010118    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7c6027\""
Oct 30 19:17:38.010258 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.010223    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c6027\"/\"payload-processing-external-model-e2e--a67c8769-dockercfg-czgjj\""
Oct 30 19:17:38.155286 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.155247    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c89796b-9ed3-4d26-a3e1-89525585be35-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:38.155536 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.155436    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c89796b-9ed3-4d26-a3e1-89525585be35-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:38.155620 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.155541    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd4d5\" (UniqueName: \"kubernetes.io/projected/9c89796b-9ed3-4d26-a3e1-89525585be35-kube-api-access-bd4d5\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:38.155683 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.155671    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c89796b-9ed3-4d26-a3e1-89525585be35-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:38.257663 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.257583    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c89796b-9ed3-4d26-a3e1-89525585be35-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:38.257832 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.257784    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c89796b-9ed3-4d26-a3e1-89525585be35-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:38.257924 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.257903    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c89796b-9ed3-4d26-a3e1-89525585be35-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:38.257988 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.257954    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4d5\" (UniqueName: \"kubernetes.io/projected/9c89796b-9ed3-4d26-a3e1-89525585be35-kube-api-access-bd4d5\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:38.258554 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.258526    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c89796b-9ed3-4d26-a3e1-89525585be35-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:38.258703 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.258639    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c89796b-9ed3-4d26-a3e1-89525585be35-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:38.261182 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.261155    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c89796b-9ed3-4d26-a3e1-89525585be35-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:38.273095 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.273071    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4d5\" (UniqueName: \"kubernetes.io/projected/9c89796b-9ed3-4d26-a3e1-89525585be35-kube-api-access-bd4d5\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:38.315251 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.315216    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:38.345216 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.345185    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"]
Oct 30 19:17:38.350181 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.350152    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:38.358580 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.358555    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"openshift-service-ca.crt\""
Oct 30 19:17:38.358925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.358901    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"payload-processing-external-model-e2e--0575c797-dockercfg-2h4h6\""
Oct 30 19:17:38.359150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.359107    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"agc-cred-cb587320c193\""
Oct 30 19:17:38.359345 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.359318    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"agc-cred-7422f6dea553\""
Oct 30 19:17:38.359454 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.359427    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"payload-processing-external-model-plugins-e2e-extmod-a-de4bda\""
Oct 30 19:17:38.359523 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.359488    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"kube-root-ca.crt\""
Oct 30 19:17:38.359523 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.359407    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"routing-overlay\""
Oct 30 19:17:38.366999 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.366973    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"]
Oct 30 19:17:38.461572 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.461540    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6faafbec-abd9-4f16-a979-e607b39565ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:38.461844 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.461822    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6faafbec-abd9-4f16-a979-e607b39565ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:38.461892 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.461860    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8shg\" (UniqueName: \"kubernetes.io/projected/6faafbec-abd9-4f16-a979-e607b39565ee-kube-api-access-s8shg\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:38.461941 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.461919    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6faafbec-abd9-4f16-a979-e607b39565ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:38.568549 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.564740    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6faafbec-abd9-4f16-a979-e607b39565ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:38.568549 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.565057    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6faafbec-abd9-4f16-a979-e607b39565ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:38.568549 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.565101    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8shg\" (UniqueName: \"kubernetes.io/projected/6faafbec-abd9-4f16-a979-e607b39565ee-kube-api-access-s8shg\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:38.568549 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.565163    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6faafbec-abd9-4f16-a979-e607b39565ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:38.568549 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.566532    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6faafbec-abd9-4f16-a979-e607b39565ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:38.568963 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.568723    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6faafbec-abd9-4f16-a979-e607b39565ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:38.594153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.573716    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6faafbec-abd9-4f16-a979-e607b39565ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:38.705151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.704959    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8shg\" (UniqueName: \"kubernetes.io/projected/6faafbec-abd9-4f16-a979-e607b39565ee-kube-api-access-s8shg\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:38.857960 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.857922    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:38.927103 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:38.927066    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"]
Oct 30 19:17:39.154648 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.154569    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"]
Oct 30 19:17:39.159687 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.159665    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:39.166742 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.166693    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"payload-processing-external-model-plugins-e2e-extmod-a-21d896\""
Oct 30 19:17:39.166902 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.166763    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"routing-overlay\""
Oct 30 19:17:39.166982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.166913    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"kube-root-ca.crt\""
Oct 30 19:17:39.167109 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.167093    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"openshift-service-ca.crt\""
Oct 30 19:17:39.172375 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.172353    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"]
Oct 30 19:17:39.187473 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.187367    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"agc-cred-676f802b0d13\""
Oct 30 19:17:39.187578 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.187522    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"payload-processing-external-model-e2e--3f19eb37-dockercfg-phgbc\""
Oct 30 19:17:39.240252 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.240228    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"]
Oct 30 19:17:39.240735 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:17:39.240713    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6faafbec_abd9_4f16_a979_e607b39565ee.slice/crio-e510104573f672bf5a8135005f5c3a750c65fdfcd0c2e3ee967d841d1161befb WatchSource:0}: Error finding container e510104573f672bf5a8135005f5c3a750c65fdfcd0c2e3ee967d841d1161befb: Status 404 returned error can't find the container with id e510104573f672bf5a8135005f5c3a750c65fdfcd0c2e3ee967d841d1161befb
Oct 30 19:17:39.264041 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.264015    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/033e092c-66f5-4b8b-adcc-541587bd779b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:39.264288 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.264267    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033e092c-66f5-4b8b-adcc-541587bd779b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:39.264342 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.264310    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94nvd\" (UniqueName: \"kubernetes.io/projected/033e092c-66f5-4b8b-adcc-541587bd779b-kube-api-access-94nvd\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:39.264458 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.264439    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/033e092c-66f5-4b8b-adcc-541587bd779b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:39.380221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.378075    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94nvd\" (UniqueName: \"kubernetes.io/projected/033e092c-66f5-4b8b-adcc-541587bd779b-kube-api-access-94nvd\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:39.380221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.378263    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/033e092c-66f5-4b8b-adcc-541587bd779b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:39.380221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.378607    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/033e092c-66f5-4b8b-adcc-541587bd779b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:39.380221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.378864    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033e092c-66f5-4b8b-adcc-541587bd779b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:39.401327 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.392778    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/033e092c-66f5-4b8b-adcc-541587bd779b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:39.401327 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.400979    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033e092c-66f5-4b8b-adcc-541587bd779b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:39.414268 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.412199    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/033e092c-66f5-4b8b-adcc-541587bd779b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:39.432434 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.425844    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94nvd\" (UniqueName: \"kubernetes.io/projected/033e092c-66f5-4b8b-adcc-541587bd779b-kube-api-access-94nvd\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:39.486039 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.485755    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:39.964198 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.963722    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc" event={"ID":"6faafbec-abd9-4f16-a979-e607b39565ee","Type":"ContainerStarted","Data":"a843d11a3d6fb2645156537a968d84c4ac9f5be879895aaf2bf0ed0e7e1f68cf"}
Oct 30 19:17:39.964198 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.963808    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc" event={"ID":"6faafbec-abd9-4f16-a979-e607b39565ee","Type":"ContainerStarted","Data":"e510104573f672bf5a8135005f5c3a750c65fdfcd0c2e3ee967d841d1161befb"}
Oct 30 19:17:39.987152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:39.964922    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:40.058427 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:40.047432    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d" event={"ID":"9c89796b-9ed3-4d26-a3e1-89525585be35","Type":"ContainerStarted","Data":"8d095bc22255e5c476b762f6e1c4e1b50265d11f2a6b36f82582790001f3192f"}
Oct 30 19:17:40.058427 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:40.047499    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d" event={"ID":"9c89796b-9ed3-4d26-a3e1-89525585be35","Type":"ContainerStarted","Data":"f90f81e2be9ef5162e9bb2cfaca62ab5a36e18294b79105c06ea203c723929e3"}
Oct 30 19:17:40.058427 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:40.048803    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:40.101286 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:40.101199    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"]
Oct 30 19:17:40.127148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:40.117937    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc" podStartSLOduration=2.117916563 podStartE2EDuration="2.117916563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:39.99998691 +0000 UTC m=+2673.868405240" watchObservedRunningTime="2026-10-30 19:17:40.117916563 +0000 UTC m=+2673.986334899"
Oct 30 19:17:40.168160 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:40.149573    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d" podStartSLOduration=3.149557758 podStartE2EDuration="3.149557758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:40.083579721 +0000 UTC m=+2673.951998051" watchObservedRunningTime="2026-10-30 19:17:40.149557758 +0000 UTC m=+2674.017976089"
Oct 30 19:17:40.415050 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:40.415022    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"]
Oct 30 19:17:40.415269 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:40.415243    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm" podUID="6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" containerName="istio-proxy" containerID="cri-o://cf9d13091e025a584ec60732fd114bd0e7252ca26a0649c0ad5c7a8dc37469fa" gracePeriod=30
Oct 30 19:17:41.056815 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.056775    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2" event={"ID":"033e092c-66f5-4b8b-adcc-541587bd779b","Type":"ContainerStarted","Data":"5ac0399f0f8cd796eab167ad76d007656b838a29b73446d8aacaceb4834e9ffd"}
Oct 30 19:17:41.057034 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.056841    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2" event={"ID":"033e092c-66f5-4b8b-adcc-541587bd779b","Type":"ContainerStarted","Data":"88abaadfb288e76541679c99b1124bd30219bf4ec1193152dd0e23f3d98e0e54"}
Oct 30 19:17:41.057034 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.056868    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:41.082514 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.082471    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2" podStartSLOduration=2.082457093 podStartE2EDuration="2.082457093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:41.078866734 +0000 UTC m=+2674.947285084" watchObservedRunningTime="2026-10-30 19:17:41.082457093 +0000 UTC m=+2674.950875420"
Oct 30 19:17:41.374043 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.374004    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"]
Oct 30 19:17:41.378938 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.378904    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:41.387865 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.386618    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"routing-overlay\""
Oct 30 19:17:41.387865 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.387030    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d72beb\""
Oct 30 19:17:41.387865 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.387036    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"openshift-service-ca.crt\""
Oct 30 19:17:41.387865 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.387105    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"agc-cred-1a4160222042\""
Oct 30 19:17:41.387865 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.387281    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"kube-root-ca.crt\""
Oct 30 19:17:41.387865 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.387596    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"payload-processing-external-model-e2e--b99c0f48-dockercfg-zjmbc\""
Oct 30 19:17:41.392637 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.392614    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"]
Oct 30 19:17:41.563152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.562488    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:41.563152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.562600    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:41.563152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.562647    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:41.563152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.563005    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctfrc\" (UniqueName: \"kubernetes.io/projected/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-kube-api-access-ctfrc\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:41.703213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.702633    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctfrc\" (UniqueName: \"kubernetes.io/projected/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-kube-api-access-ctfrc\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:41.703213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.702828    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:41.703213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.702934    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:41.703213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.702981    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:41.719157 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.704025    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:41.719157 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.705350    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:41.756775 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.754767    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:41.778155 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.773771    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctfrc\" (UniqueName: \"kubernetes.io/projected/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-kube-api-access-ctfrc\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:42.000917 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:41.995797    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:42.298517 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:17:42.298487    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3325ec4_99b3_485c_9119_b0e9a8c61ffb.slice/crio-21683ef3b06fdaefa04eba85f1bc06cb213309da8b507b19049a222d3d644c57 WatchSource:0}: Error finding container 21683ef3b06fdaefa04eba85f1bc06cb213309da8b507b19049a222d3d644c57: Status 404 returned error can't find the container with id 21683ef3b06fdaefa04eba85f1bc06cb213309da8b507b19049a222d3d644c57
Oct 30 19:17:42.298607 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:42.298584    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"]
Oct 30 19:17:43.082209 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:43.082173    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99" event={"ID":"d3325ec4-99b3-485c-9119-b0e9a8c61ffb","Type":"ContainerStarted","Data":"6edb454503f574c6cb7bfb5578b7cbc29777c97dc67b13b2b14a30c7405d2937"}
Oct 30 19:17:43.082619 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:43.082235    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99" event={"ID":"d3325ec4-99b3-485c-9119-b0e9a8c61ffb","Type":"ContainerStarted","Data":"21683ef3b06fdaefa04eba85f1bc06cb213309da8b507b19049a222d3d644c57"}
Oct 30 19:17:43.082619 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:43.082276    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:43.103970 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:43.103930    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99" podStartSLOduration=2.103920145 podStartE2EDuration="2.103920145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:43.103641383 +0000 UTC m=+2676.972059713" watchObservedRunningTime="2026-10-30 19:17:43.103920145 +0000 UTC m=+2676.972338474"
Oct 30 19:17:45.733507 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.733449    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:17:45.839583 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.839542    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-credential-socket\") pod \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") "
Oct 30 19:17:45.839799 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.839636    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-workload-certs\") pod \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") "
Oct 30 19:17:45.839799 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.839689    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-podinfo\") pod \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") "
Oct 30 19:17:45.839799 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.839788    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-data\") pod \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") "
Oct 30 19:17:45.840001 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.839847    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istiod-ca-cert\") pod \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") "
Oct 30 19:17:45.840001 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.839858    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-credential-socket" pod "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" (UID: "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:45.840001 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.839886    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-workload-socket\") pod \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") "
Oct 30 19:17:45.840001 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.839917    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-kube-api-access-xgzr8\" (UniqueName: \"kubernetes.io/projected/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-kube-api-access-xgzr8\") pod \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") "
Oct 30 19:17:45.840255 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.840049    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-data" pod "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" (UID: "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:45.840255 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.840065    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-token\" (UniqueName: \"kubernetes.io/projected/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-token\") pod \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") "
Oct 30 19:17:45.840255 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.840172    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-envoy\") pod \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\" (UID: \"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c\") "
Oct 30 19:17:45.840423 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.840305    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-workload-certs" pod "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" (UID: "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:45.840745 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.840645    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-workload-socket" pod "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" (UID: "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:45.841454 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.841429    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:45.841592 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.841578    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:45.841687 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.841675    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:45.841810 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.841796    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:45.841906 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.841459    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istiod-ca-cert" pod "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" (UID: "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:45.843205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.843175    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-podinfo" pod "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" (UID: "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:45.844335 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.844301    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-envoy" pod "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" (UID: "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:45.845992 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.845963    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-kube-api-access-xgzr8" pod "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" (UID: "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c"). InnerVolumeSpecName "kube-api-access-xgzr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:45.847485 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.847459    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-token" pod "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" (UID: "6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:45.944038 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.944003    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:45.944038 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.944036    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:45.944283 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.944046    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgzr8\" (UniqueName: \"kubernetes.io/projected/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-kube-api-access-xgzr8\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:45.944283 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.944057    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:45.944283 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:45.944065    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.108034 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.108001    4372 generic.go:356] "Generic (PLEG): container finished" podID="6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" containerID="cf9d13091e025a584ec60732fd114bd0e7252ca26a0649c0ad5c7a8dc37469fa" exitCode=0
Oct 30 19:17:46.108329 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.108060    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"
Oct 30 19:17:46.108329 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.108086    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm" event={"ID":"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c","Type":"ContainerDied","Data":"cf9d13091e025a584ec60732fd114bd0e7252ca26a0649c0ad5c7a8dc37469fa"}
Oct 30 19:17:46.108329 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.108197    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm" event={"ID":"6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c","Type":"ContainerDied","Data":"a4514d1e4c4cc2984d70aa6d3863794a40f29ac8c3cc2cc46ba6f0a359e61c01"}
Oct 30 19:17:46.108329 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.108238    4372 scope.go:122] "RemoveContainer" containerID="cf9d13091e025a584ec60732fd114bd0e7252ca26a0649c0ad5c7a8dc37469fa"
Oct 30 19:17:46.120871 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.120851    4372 scope.go:122] "RemoveContainer" containerID="cf9d13091e025a584ec60732fd114bd0e7252ca26a0649c0ad5c7a8dc37469fa"
Oct 30 19:17:46.121167 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:17:46.121111    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf9d13091e025a584ec60732fd114bd0e7252ca26a0649c0ad5c7a8dc37469fa\": container with ID starting with cf9d13091e025a584ec60732fd114bd0e7252ca26a0649c0ad5c7a8dc37469fa not found: ID does not exist" containerID="cf9d13091e025a584ec60732fd114bd0e7252ca26a0649c0ad5c7a8dc37469fa"
Oct 30 19:17:46.121244 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.121175    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9d13091e025a584ec60732fd114bd0e7252ca26a0649c0ad5c7a8dc37469fa"} err="failed to get container status \"cf9d13091e025a584ec60732fd114bd0e7252ca26a0649c0ad5c7a8dc37469fa\": rpc error: code = NotFound desc = could not find container \"cf9d13091e025a584ec60732fd114bd0e7252ca26a0649c0ad5c7a8dc37469fa\": container with ID starting with cf9d13091e025a584ec60732fd114bd0e7252ca26a0649c0ad5c7a8dc37469fa not found: ID does not exist"
Oct 30 19:17:46.134992 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.134969    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"]
Oct 30 19:17:46.135816 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.135801    4372 cpu_manager.go:414] "containerMap: removing container" podUID="6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" containerName="istio-proxy"
Oct 30 19:17:46.135874 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.135819    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" containerName="istio-proxy"
Oct 30 19:17:46.135945 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.135936    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" containerName="istio-proxy"
Oct 30 19:17:46.141403 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.141386    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:46.142064 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.142044    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"]
Oct 30 19:17:46.152350 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.152081    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"agc-cred-919331a7b490\""
Oct 30 19:17:46.155626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.155606    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-x4dbm"]
Oct 30 19:17:46.160534 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.160515    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"]
Oct 30 19:17:46.248548 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.248517    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88adeb15-521e-4bf1-85bd-6b549ea5c460-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:46.248726 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.248705    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:46.248803 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.248790    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwtfq\" (UniqueName: \"kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-kube-api-access-zwtfq\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:46.248995 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.248974    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88adeb15-521e-4bf1-85bd-6b549ea5c460-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:46.268196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.268173    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c" path="/var/lib/kubelet/pods/6c5bee02-c4e2-4043-a111-f5d3d6a3cc4c/volumes"
Oct 30 19:17:46.350289 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.350259    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:46.350438 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.350332    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwtfq\" (UniqueName: \"kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-kube-api-access-zwtfq\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:46.350557 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.350541    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88adeb15-521e-4bf1-85bd-6b549ea5c460-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:46.350626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.350588    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88adeb15-521e-4bf1-85bd-6b549ea5c460-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:46.351243 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.351220    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88adeb15-521e-4bf1-85bd-6b549ea5c460-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:46.351357 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.351344    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88adeb15-521e-4bf1-85bd-6b549ea5c460-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:46.353070 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.353044    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:46.366392 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.366343    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwtfq\" (UniqueName: \"kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-kube-api-access-zwtfq\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:46.456500 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.456477    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:46.797724 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:17:46.797686    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88adeb15_521e_4bf1_85bd_6b549ea5c460.slice/crio-0f9fee5488c327bdeb4b3543b9935d5c438bdb67b67fa8151956ddca7c4fd8dc WatchSource:0}: Error finding container 0f9fee5488c327bdeb4b3543b9935d5c438bdb67b67fa8151956ddca7c4fd8dc: Status 404 returned error can't find the container with id 0f9fee5488c327bdeb4b3543b9935d5c438bdb67b67fa8151956ddca7c4fd8dc
Oct 30 19:17:46.800907 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:46.800887    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"]
Oct 30 19:17:47.117843 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:47.117801    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl" event={"ID":"88adeb15-521e-4bf1-85bd-6b549ea5c460","Type":"ContainerStarted","Data":"6d9248dc3acd972750a037253e9985f457c15fb18c164fccfd4b2eab3f3ecbed"}
Oct 30 19:17:47.117992 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:47.117873    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl" event={"ID":"88adeb15-521e-4bf1-85bd-6b549ea5c460","Type":"ContainerStarted","Data":"0f9fee5488c327bdeb4b3543b9935d5c438bdb67b67fa8151956ddca7c4fd8dc"}
Oct 30 19:17:47.118065 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:47.118021    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:47.147383 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:47.147341    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl" podStartSLOduration=1.147329246 podStartE2EDuration="1.147329246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:47.146019957 +0000 UTC m=+2681.014438300" watchObservedRunningTime="2026-10-30 19:17:47.147329246 +0000 UTC m=+2681.015747572"
Oct 30 19:17:52.076475 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:52.076439    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:17:52.077348 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:52.077321    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:17:52.077501 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:52.077487    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:17:54.093368 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:54.093340    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:17:57.766716 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:17:57.766672    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:17:57.912508 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:57.912465    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"]
Oct 30 19:17:58.130738 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:58.130705    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:17:58.209289 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:17:58.209252    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"]
Oct 30 19:18:02.913399 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:02.913355    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2" podUID="033e092c-66f5-4b8b-adcc-541587bd779b" containerName="payload-processing" containerID="cri-o://5ac0399f0f8cd796eab167ad76d007656b838a29b73446d8aacaceb4834e9ffd" gracePeriod=25
Oct 30 19:18:03.209738 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.209714    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:18:03.209872 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.209710    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99" podUID="d3325ec4-99b3-485c-9119-b0e9a8c61ffb" containerName="payload-processing" containerID="cri-o://6edb454503f574c6cb7bfb5578b7cbc29777c97dc67b13b2b14a30c7405d2937" gracePeriod=25
Oct 30 19:18:03.300823 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.300783    4372 generic.go:356] "Generic (PLEG): container finished" podID="033e092c-66f5-4b8b-adcc-541587bd779b" containerID="5ac0399f0f8cd796eab167ad76d007656b838a29b73446d8aacaceb4834e9ffd" exitCode=0
Oct 30 19:18:03.300997 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.300887    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2" event={"ID":"033e092c-66f5-4b8b-adcc-541587bd779b","Type":"ContainerDied","Data":"5ac0399f0f8cd796eab167ad76d007656b838a29b73446d8aacaceb4834e9ffd"}
Oct 30 19:18:03.300997 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.300936    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2" event={"ID":"033e092c-66f5-4b8b-adcc-541587bd779b","Type":"ContainerDied","Data":"88abaadfb288e76541679c99b1124bd30219bf4ec1193152dd0e23f3d98e0e54"}
Oct 30 19:18:03.300997 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.300965    4372 scope.go:122] "RemoveContainer" containerID="5ac0399f0f8cd796eab167ad76d007656b838a29b73446d8aacaceb4834e9ffd"
Oct 30 19:18:03.301260 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.301147    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"
Oct 30 19:18:03.325800 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.325766    4372 scope.go:122] "RemoveContainer" containerID="5ac0399f0f8cd796eab167ad76d007656b838a29b73446d8aacaceb4834e9ffd"
Oct 30 19:18:03.326119 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:03.326090    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac0399f0f8cd796eab167ad76d007656b838a29b73446d8aacaceb4834e9ffd\": container with ID starting with 5ac0399f0f8cd796eab167ad76d007656b838a29b73446d8aacaceb4834e9ffd not found: ID does not exist" containerID="5ac0399f0f8cd796eab167ad76d007656b838a29b73446d8aacaceb4834e9ffd"
Oct 30 19:18:03.326215 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.326174    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac0399f0f8cd796eab167ad76d007656b838a29b73446d8aacaceb4834e9ffd"} err="failed to get container status \"5ac0399f0f8cd796eab167ad76d007656b838a29b73446d8aacaceb4834e9ffd\": rpc error: code = NotFound desc = could not find container \"5ac0399f0f8cd796eab167ad76d007656b838a29b73446d8aacaceb4834e9ffd\": container with ID starting with 5ac0399f0f8cd796eab167ad76d007656b838a29b73446d8aacaceb4834e9ffd not found: ID does not exist"
Oct 30 19:18:03.330986 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.330961    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033e092c-66f5-4b8b-adcc-541587bd779b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/033e092c-66f5-4b8b-adcc-541587bd779b-provider-credentials\") pod \"033e092c-66f5-4b8b-adcc-541587bd779b\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") "
Oct 30 19:18:03.331176 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.331154    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033e092c-66f5-4b8b-adcc-541587bd779b-kube-api-access-94nvd\" (UniqueName: \"kubernetes.io/projected/033e092c-66f5-4b8b-adcc-541587bd779b-kube-api-access-94nvd\") pod \"033e092c-66f5-4b8b-adcc-541587bd779b\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") "
Oct 30 19:18:03.331300 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.331191    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033e092c-66f5-4b8b-adcc-541587bd779b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033e092c-66f5-4b8b-adcc-541587bd779b-plugins-config-volume\") pod \"033e092c-66f5-4b8b-adcc-541587bd779b\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") "
Oct 30 19:18:03.331300 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.331268    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033e092c-66f5-4b8b-adcc-541587bd779b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/033e092c-66f5-4b8b-adcc-541587bd779b-routing-overlay-volume\") pod \"033e092c-66f5-4b8b-adcc-541587bd779b\" (UID: \"033e092c-66f5-4b8b-adcc-541587bd779b\") "
Oct 30 19:18:03.331680 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.331653    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033e092c-66f5-4b8b-adcc-541587bd779b-plugins-config-volume" pod "033e092c-66f5-4b8b-adcc-541587bd779b" (UID: "033e092c-66f5-4b8b-adcc-541587bd779b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:03.331773 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.331658    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033e092c-66f5-4b8b-adcc-541587bd779b-routing-overlay-volume" pod "033e092c-66f5-4b8b-adcc-541587bd779b" (UID: "033e092c-66f5-4b8b-adcc-541587bd779b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:03.332100 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.332079    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033e092c-66f5-4b8b-adcc-541587bd779b-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:03.332100 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.332102    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/033e092c-66f5-4b8b-adcc-541587bd779b-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:03.334097 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.334074    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033e092c-66f5-4b8b-adcc-541587bd779b-kube-api-access-94nvd" pod "033e092c-66f5-4b8b-adcc-541587bd779b" (UID: "033e092c-66f5-4b8b-adcc-541587bd779b"). InnerVolumeSpecName "kube-api-access-94nvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:03.334201 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.334110    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033e092c-66f5-4b8b-adcc-541587bd779b-provider-credentials" pod "033e092c-66f5-4b8b-adcc-541587bd779b" (UID: "033e092c-66f5-4b8b-adcc-541587bd779b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:03.437227 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.434962    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/033e092c-66f5-4b8b-adcc-541587bd779b-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:03.437227 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.435102    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94nvd\" (UniqueName: \"kubernetes.io/projected/033e092c-66f5-4b8b-adcc-541587bd779b-kube-api-access-94nvd\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:03.560774 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.560746    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:18:03.632736 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.632706    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"]
Oct 30 19:18:03.638526 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.638499    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-plugins-config-volume\") pod \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") "
Oct 30 19:18:03.638701 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.638685    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-routing-overlay-volume\") pod \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") "
Oct 30 19:18:03.638884 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.638868    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-kube-api-access-ctfrc\" (UniqueName: \"kubernetes.io/projected/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-kube-api-access-ctfrc\") pod \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") "
Oct 30 19:18:03.638980 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.638951    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-provider-credentials\") pod \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\" (UID: \"d3325ec4-99b3-485c-9119-b0e9a8c61ffb\") "
Oct 30 19:18:03.639042 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.638990    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-plugins-config-volume" pod "d3325ec4-99b3-485c-9119-b0e9a8c61ffb" (UID: "d3325ec4-99b3-485c-9119-b0e9a8c61ffb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:03.639042 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.639032    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-routing-overlay-volume" pod "d3325ec4-99b3-485c-9119-b0e9a8c61ffb" (UID: "d3325ec4-99b3-485c-9119-b0e9a8c61ffb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:03.639693 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.639665    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8ffqtg2"]
Oct 30 19:18:03.639917 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.639688    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:03.639917 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.639713    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:03.641422 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.641399    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-provider-credentials" pod "d3325ec4-99b3-485c-9119-b0e9a8c61ffb" (UID: "d3325ec4-99b3-485c-9119-b0e9a8c61ffb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:03.641510 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.641432    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-kube-api-access-ctfrc" pod "d3325ec4-99b3-485c-9119-b0e9a8c61ffb" (UID: "d3325ec4-99b3-485c-9119-b0e9a8c61ffb"). InnerVolumeSpecName "kube-api-access-ctfrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:03.741171 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.741061    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctfrc\" (UniqueName: \"kubernetes.io/projected/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-kube-api-access-ctfrc\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:03.741171 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:03.741093    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3325ec4-99b3-485c-9119-b0e9a8c61ffb-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:04.268669 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:04.268640    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033e092c-66f5-4b8b-adcc-541587bd779b" path="/var/lib/kubelet/pods/033e092c-66f5-4b8b-adcc-541587bd779b/volumes"
Oct 30 19:18:04.309042 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:04.309015    4372 generic.go:356] "Generic (PLEG): container finished" podID="d3325ec4-99b3-485c-9119-b0e9a8c61ffb" containerID="6edb454503f574c6cb7bfb5578b7cbc29777c97dc67b13b2b14a30c7405d2937" exitCode=0
Oct 30 19:18:04.309197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:04.309057    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"
Oct 30 19:18:04.309197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:04.309100    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99" event={"ID":"d3325ec4-99b3-485c-9119-b0e9a8c61ffb","Type":"ContainerDied","Data":"6edb454503f574c6cb7bfb5578b7cbc29777c97dc67b13b2b14a30c7405d2937"}
Oct 30 19:18:04.309197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:04.309165    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99" event={"ID":"d3325ec4-99b3-485c-9119-b0e9a8c61ffb","Type":"ContainerDied","Data":"21683ef3b06fdaefa04eba85f1bc06cb213309da8b507b19049a222d3d644c57"}
Oct 30 19:18:04.309197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:04.309187    4372 scope.go:122] "RemoveContainer" containerID="6edb454503f574c6cb7bfb5578b7cbc29777c97dc67b13b2b14a30c7405d2937"
Oct 30 19:18:04.319715 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:04.319685    4372 scope.go:122] "RemoveContainer" containerID="6edb454503f574c6cb7bfb5578b7cbc29777c97dc67b13b2b14a30c7405d2937"
Oct 30 19:18:04.319991 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:04.319969    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6edb454503f574c6cb7bfb5578b7cbc29777c97dc67b13b2b14a30c7405d2937\": container with ID starting with 6edb454503f574c6cb7bfb5578b7cbc29777c97dc67b13b2b14a30c7405d2937 not found: ID does not exist" containerID="6edb454503f574c6cb7bfb5578b7cbc29777c97dc67b13b2b14a30c7405d2937"
Oct 30 19:18:04.320068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:04.320001    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6edb454503f574c6cb7bfb5578b7cbc29777c97dc67b13b2b14a30c7405d2937"} err="failed to get container status \"6edb454503f574c6cb7bfb5578b7cbc29777c97dc67b13b2b14a30c7405d2937\": rpc error: code = NotFound desc = could not find container \"6edb454503f574c6cb7bfb5578b7cbc29777c97dc67b13b2b14a30c7405d2937\": container with ID starting with 6edb454503f574c6cb7bfb5578b7cbc29777c97dc67b13b2b14a30c7405d2937 not found: ID does not exist"
Oct 30 19:18:04.333675 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:04.333649    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"]
Oct 30 19:18:04.338059 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:04.338040    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65kxb99"]
Oct 30 19:18:06.269493 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:06.269432    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3325ec4-99b3-485c-9119-b0e9a8c61ffb" path="/var/lib/kubelet/pods/d3325ec4-99b3-485c-9119-b0e9a8c61ffb/volumes"
Oct 30 19:18:27.385397 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:27.385358    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:18:27.697598 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:27.697517    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:18:46.221149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.216190    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"]
Oct 30 19:18:46.221149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.217501    4372 cpu_manager.go:414] "containerMap: removing container" podUID="033e092c-66f5-4b8b-adcc-541587bd779b" containerName="payload-processing"
Oct 30 19:18:46.221149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.217525    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033e092c-66f5-4b8b-adcc-541587bd779b" containerName="payload-processing"
Oct 30 19:18:46.221149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.217558    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d3325ec4-99b3-485c-9119-b0e9a8c61ffb" containerName="payload-processing"
Oct 30 19:18:46.221149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.217568    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3325ec4-99b3-485c-9119-b0e9a8c61ffb" containerName="payload-processing"
Oct 30 19:18:46.221149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.217777    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="033e092c-66f5-4b8b-adcc-541587bd779b" containerName="payload-processing"
Oct 30 19:18:46.221149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.217796    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3325ec4-99b3-485c-9119-b0e9a8c61ffb" containerName="payload-processing"
Oct 30 19:18:46.234885 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.234858    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:46.244815 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.243462    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 19:18:46.244815 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.244193    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fee8e82c3bd8\""
Oct 30 19:18:46.264149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.251013    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-wclh5\""
Oct 30 19:18:46.264149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.251093    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 19:18:46.264149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.251204    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 19:18:46.264149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.251286    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 19:18:46.399178 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.398085    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-provider-credentials\") pod \"payload-processing-external-model-75cb56d5b9-ntwxl\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") " pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:46.399178 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.398164    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7tt5\" (UniqueName: \"kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-kube-api-access-m7tt5\") pod \"payload-processing-external-model-75cb56d5b9-ntwxl\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") " pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:46.399178 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.398252    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74d7f977-902a-48ea-870d-08712e64e26a-routing-overlay-volume\") pod \"payload-processing-external-model-75cb56d5b9-ntwxl\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") " pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:46.399178 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.398312    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74d7f977-902a-48ea-870d-08712e64e26a-plugins-config-volume\") pod \"payload-processing-external-model-75cb56d5b9-ntwxl\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") " pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:46.406460 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.406083    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-8348b0e5ef0b\""
Oct 30 19:18:46.421254 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.421218    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"]
Oct 30 19:18:46.499478 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.499388    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-provider-credentials\") pod \"payload-processing-external-model-75cb56d5b9-ntwxl\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") " pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:46.499478 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.499453    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7tt5\" (UniqueName: \"kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-kube-api-access-m7tt5\") pod \"payload-processing-external-model-75cb56d5b9-ntwxl\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") " pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:46.499823 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.499793    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74d7f977-902a-48ea-870d-08712e64e26a-routing-overlay-volume\") pod \"payload-processing-external-model-75cb56d5b9-ntwxl\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") " pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:46.499947 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.499929    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74d7f977-902a-48ea-870d-08712e64e26a-plugins-config-volume\") pod \"payload-processing-external-model-75cb56d5b9-ntwxl\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") " pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:46.500567 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.500535    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74d7f977-902a-48ea-870d-08712e64e26a-routing-overlay-volume\") pod \"payload-processing-external-model-75cb56d5b9-ntwxl\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") " pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:46.500836 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.500690    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74d7f977-902a-48ea-870d-08712e64e26a-plugins-config-volume\") pod \"payload-processing-external-model-75cb56d5b9-ntwxl\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") " pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:46.502683 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.502661    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-provider-credentials\") pod \"payload-processing-external-model-75cb56d5b9-ntwxl\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") " pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:46.512761 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.512741    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7tt5\" (UniqueName: \"kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-kube-api-access-m7tt5\") pod \"payload-processing-external-model-75cb56d5b9-ntwxl\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") " pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:46.625161 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:46.620455    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:47.173484 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.173451    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"]
Oct 30 19:18:47.174423 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:18:47.174393    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d7f977_902a_48ea_870d_08712e64e26a.slice/crio-0553cca44c74bd27cd2ab6e2ba838d190dc0b7b99e27041fc2cc0e452a4075d4 WatchSource:0}: Error finding container 0553cca44c74bd27cd2ab6e2ba838d190dc0b7b99e27041fc2cc0e452a4075d4: Status 404 returned error can't find the container with id 0553cca44c74bd27cd2ab6e2ba838d190dc0b7b99e27041fc2cc0e452a4075d4
Oct 30 19:18:47.573355 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.567173    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"]
Oct 30 19:18:47.616149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.616013    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"]
Oct 30 19:18:47.643666 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.643635    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"]
Oct 30 19:18:47.644089 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.644021    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:47.726339 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.726199    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e4dbe81-f519-4420-9611-987f44aafd42-routing-overlay-volume\") pod \"payload-processing-external-model-865f6676c9-n76w2\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") " pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:47.751153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.726972    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e4dbe81-f519-4420-9611-987f44aafd42-provider-credentials\") pod \"payload-processing-external-model-865f6676c9-n76w2\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") " pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:47.751533 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.751496    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4dbe81-f519-4420-9611-987f44aafd42-plugins-config-volume\") pod \"payload-processing-external-model-865f6676c9-n76w2\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") " pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:47.752001 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.751975    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr4sn\" (UniqueName: \"kubernetes.io/projected/8e4dbe81-f519-4420-9611-987f44aafd42-kube-api-access-tr4sn\") pod \"payload-processing-external-model-865f6676c9-n76w2\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") " pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:47.867150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.858285    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e4dbe81-f519-4420-9611-987f44aafd42-routing-overlay-volume\") pod \"payload-processing-external-model-865f6676c9-n76w2\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") " pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:47.867150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.858381    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e4dbe81-f519-4420-9611-987f44aafd42-provider-credentials\") pod \"payload-processing-external-model-865f6676c9-n76w2\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") " pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:47.867150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.858441    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4dbe81-f519-4420-9611-987f44aafd42-plugins-config-volume\") pod \"payload-processing-external-model-865f6676c9-n76w2\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") " pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:47.867150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.858710    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr4sn\" (UniqueName: \"kubernetes.io/projected/8e4dbe81-f519-4420-9611-987f44aafd42-kube-api-access-tr4sn\") pod \"payload-processing-external-model-865f6676c9-n76w2\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") " pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:47.867150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.860388    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e4dbe81-f519-4420-9611-987f44aafd42-routing-overlay-volume\") pod \"payload-processing-external-model-865f6676c9-n76w2\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") " pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:47.867150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.862186    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4dbe81-f519-4420-9611-987f44aafd42-plugins-config-volume\") pod \"payload-processing-external-model-865f6676c9-n76w2\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") " pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:47.867150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.862983    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl" event={"ID":"74d7f977-902a-48ea-870d-08712e64e26a","Type":"ContainerStarted","Data":"1fd546181b940038fb1389a31d621ed22e2721e88243a7daaca0d86c3d9f4a06"}
Oct 30 19:18:47.867150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.863046    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl" event={"ID":"74d7f977-902a-48ea-870d-08712e64e26a","Type":"ContainerStarted","Data":"0553cca44c74bd27cd2ab6e2ba838d190dc0b7b99e27041fc2cc0e452a4075d4"}
Oct 30 19:18:47.867150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.863994    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:47.916665 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.916588    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl" podStartSLOduration=1.91656679 podStartE2EDuration="1.91656679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:47.907362514 +0000 UTC m=+2741.775780845" watchObservedRunningTime="2026-10-30 19:18:47.91656679 +0000 UTC m=+2741.784985122"
Oct 30 19:18:47.917170 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.917139    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr4sn\" (UniqueName: \"kubernetes.io/projected/8e4dbe81-f519-4420-9611-987f44aafd42-kube-api-access-tr4sn\") pod \"payload-processing-external-model-865f6676c9-n76w2\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") " pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:47.920666 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.920635    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e4dbe81-f519-4420-9611-987f44aafd42-provider-credentials\") pod \"payload-processing-external-model-865f6676c9-n76w2\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") " pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:47.962518 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:47.962482    4372 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-fee8e82c3bd8: secret "agc-cred-fee8e82c3bd8" not found
Oct 30 19:18:47.962518 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:47.962515    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl: secret "agc-cred-fee8e82c3bd8" not found
Oct 30 19:18:47.962793 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:47.962572    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-provider-credentials podName:74d7f977-902a-48ea-870d-08712e64e26a nodeName:}" failed. No retries permitted until 2026-10-30 19:18:48.462556164 +0000 UTC m=+2742.330974471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-provider-credentials") pod "payload-processing-external-model-75cb56d5b9-ntwxl" (UID: "74d7f977-902a-48ea-870d-08712e64e26a") : secret "agc-cred-fee8e82c3bd8" not found
Oct 30 19:18:47.974399 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:47.974350    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:48.301073 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:18:48.301043    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4dbe81_f519_4420_9611_987f44aafd42.slice/crio-82b1a9b71de2d391aca177f0adbf93d92ee3e77e253f3b36c84f8ed4fae6b05a WatchSource:0}: Error finding container 82b1a9b71de2d391aca177f0adbf93d92ee3e77e253f3b36c84f8ed4fae6b05a: Status 404 returned error can't find the container with id 82b1a9b71de2d391aca177f0adbf93d92ee3e77e253f3b36c84f8ed4fae6b05a
Oct 30 19:18:48.301909 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:48.301877    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"]
Oct 30 19:18:48.485944 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:48.485910    4372 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-fee8e82c3bd8: secret "agc-cred-fee8e82c3bd8" not found
Oct 30 19:18:48.485944 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:48.485941    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl: secret "agc-cred-fee8e82c3bd8" not found
Oct 30 19:18:48.486104 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:48.486000    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-provider-credentials podName:74d7f977-902a-48ea-870d-08712e64e26a nodeName:}" failed. No retries permitted until 2026-10-30 19:18:49.485982321 +0000 UTC m=+2743.354400644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-provider-credentials") pod "payload-processing-external-model-75cb56d5b9-ntwxl" (UID: "74d7f977-902a-48ea-870d-08712e64e26a") : secret "agc-cred-fee8e82c3bd8" not found
Oct 30 19:18:48.874499 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:48.874462    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2" event={"ID":"8e4dbe81-f519-4420-9611-987f44aafd42","Type":"ContainerStarted","Data":"baeff3c5bd324732ae4b4a9605da9c59cc6883f405117d4831451851110a78de"}
Oct 30 19:18:48.874950 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:48.874525    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2" event={"ID":"8e4dbe81-f519-4420-9611-987f44aafd42","Type":"ContainerStarted","Data":"82b1a9b71de2d391aca177f0adbf93d92ee3e77e253f3b36c84f8ed4fae6b05a"}
Oct 30 19:18:48.874950 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:48.874603    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:48.931000 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:48.930947    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2" podStartSLOduration=1.9309339479999998 podStartE2EDuration="1.930933948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:48.925810452 +0000 UTC m=+2742.794228781" watchObservedRunningTime="2026-10-30 19:18:48.930933948 +0000 UTC m=+2742.799352278"
Oct 30 19:18:49.514737 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:49.514702    4372 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-fee8e82c3bd8: secret "agc-cred-fee8e82c3bd8" not found
Oct 30 19:18:49.514737 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:49.514731    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl: secret "agc-cred-fee8e82c3bd8" not found
Oct 30 19:18:49.514969 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:49.514786    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-provider-credentials podName:74d7f977-902a-48ea-870d-08712e64e26a nodeName:}" failed. No retries permitted until 2026-10-30 19:18:51.514769618 +0000 UTC m=+2745.383187952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-provider-credentials") pod "payload-processing-external-model-75cb56d5b9-ntwxl" (UID: "74d7f977-902a-48ea-870d-08712e64e26a") : secret "agc-cred-fee8e82c3bd8" not found
Oct 30 19:18:51.628150 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:51.625853    4372 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-fee8e82c3bd8: secret "agc-cred-fee8e82c3bd8" not found
Oct 30 19:18:51.628150 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:51.625891    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl: secret "agc-cred-fee8e82c3bd8" not found
Oct 30 19:18:51.628150 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:51.625952    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-provider-credentials podName:74d7f977-902a-48ea-870d-08712e64e26a nodeName:}" failed. No retries permitted until 2026-10-30 19:18:55.625933417 +0000 UTC m=+2749.494351762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-provider-credentials") pod "payload-processing-external-model-75cb56d5b9-ntwxl" (UID: "74d7f977-902a-48ea-870d-08712e64e26a") : secret "agc-cred-fee8e82c3bd8" not found
Oct 30 19:18:53.875386 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:53.875298    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl" podUID="74d7f977-902a-48ea-870d-08712e64e26a" containerName="payload-processing" containerID="cri-o://1fd546181b940038fb1389a31d621ed22e2721e88243a7daaca0d86c3d9f4a06" gracePeriod=25
Oct 30 19:18:54.162977 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.162949    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:54.272790 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.272755    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74d7f977-902a-48ea-870d-08712e64e26a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74d7f977-902a-48ea-870d-08712e64e26a-plugins-config-volume\") pod \"74d7f977-902a-48ea-870d-08712e64e26a\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") "
Oct 30 19:18:54.272998 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.272899    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-kube-api-access-m7tt5\" (UniqueName: \"kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-kube-api-access-m7tt5\") pod \"74d7f977-902a-48ea-870d-08712e64e26a\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") "
Oct 30 19:18:54.273077 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.273007    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74d7f977-902a-48ea-870d-08712e64e26a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74d7f977-902a-48ea-870d-08712e64e26a-routing-overlay-volume\") pod \"74d7f977-902a-48ea-870d-08712e64e26a\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") "
Oct 30 19:18:54.273190 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.273165    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-provider-credentials\") pod \"74d7f977-902a-48ea-870d-08712e64e26a\" (UID: \"74d7f977-902a-48ea-870d-08712e64e26a\") "
Oct 30 19:18:54.273703 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.273665    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d7f977-902a-48ea-870d-08712e64e26a-routing-overlay-volume" pod "74d7f977-902a-48ea-870d-08712e64e26a" (UID: "74d7f977-902a-48ea-870d-08712e64e26a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:54.274030 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.274001    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74d7f977-902a-48ea-870d-08712e64e26a-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:54.274357 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.274326    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d7f977-902a-48ea-870d-08712e64e26a-plugins-config-volume" pod "74d7f977-902a-48ea-870d-08712e64e26a" (UID: "74d7f977-902a-48ea-870d-08712e64e26a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:54.276529 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.276501    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-kube-api-access-m7tt5" pod "74d7f977-902a-48ea-870d-08712e64e26a" (UID: "74d7f977-902a-48ea-870d-08712e64e26a"). InnerVolumeSpecName "kube-api-access-m7tt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:54.276684 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.276659    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-provider-credentials" pod "74d7f977-902a-48ea-870d-08712e64e26a" (UID: "74d7f977-902a-48ea-870d-08712e64e26a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:54.376004 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.375951    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7tt5\" (UniqueName: \"kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-kube-api-access-m7tt5\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:54.376004 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.375982    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74d7f977-902a-48ea-870d-08712e64e26a-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:54.376004 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.375998    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74d7f977-902a-48ea-870d-08712e64e26a-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:54.965674 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.965641    4372 generic.go:356] "Generic (PLEG): container finished" podID="74d7f977-902a-48ea-870d-08712e64e26a" containerID="1fd546181b940038fb1389a31d621ed22e2721e88243a7daaca0d86c3d9f4a06" exitCode=0
Oct 30 19:18:54.966195 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.965685    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"
Oct 30 19:18:54.966195 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.965728    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl" event={"ID":"74d7f977-902a-48ea-870d-08712e64e26a","Type":"ContainerDied","Data":"1fd546181b940038fb1389a31d621ed22e2721e88243a7daaca0d86c3d9f4a06"}
Oct 30 19:18:54.966195 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.965784    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl" event={"ID":"74d7f977-902a-48ea-870d-08712e64e26a","Type":"ContainerDied","Data":"0553cca44c74bd27cd2ab6e2ba838d190dc0b7b99e27041fc2cc0e452a4075d4"}
Oct 30 19:18:54.966195 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.965814    4372 scope.go:122] "RemoveContainer" containerID="1fd546181b940038fb1389a31d621ed22e2721e88243a7daaca0d86c3d9f4a06"
Oct 30 19:18:54.977988 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.977965    4372 scope.go:122] "RemoveContainer" containerID="1fd546181b940038fb1389a31d621ed22e2721e88243a7daaca0d86c3d9f4a06"
Oct 30 19:18:54.978365 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:54.978327    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd546181b940038fb1389a31d621ed22e2721e88243a7daaca0d86c3d9f4a06\": container with ID starting with 1fd546181b940038fb1389a31d621ed22e2721e88243a7daaca0d86c3d9f4a06 not found: ID does not exist" containerID="1fd546181b940038fb1389a31d621ed22e2721e88243a7daaca0d86c3d9f4a06"
Oct 30 19:18:54.978476 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.978364    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd546181b940038fb1389a31d621ed22e2721e88243a7daaca0d86c3d9f4a06"} err="failed to get container status \"1fd546181b940038fb1389a31d621ed22e2721e88243a7daaca0d86c3d9f4a06\": rpc error: code = NotFound desc = could not find container \"1fd546181b940038fb1389a31d621ed22e2721e88243a7daaca0d86c3d9f4a06\": container with ID starting with 1fd546181b940038fb1389a31d621ed22e2721e88243a7daaca0d86c3d9f4a06 not found: ID does not exist"
Oct 30 19:18:54.995036 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:54.995015    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"]
Oct 30 19:18:55.001242 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:55.001222    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cb56d5b9-ntwxl"]
Oct 30 19:18:56.328092 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:56.328053    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74d7f977-902a-48ea-870d-08712e64e26a" path="/var/lib/kubelet/pods/74d7f977-902a-48ea-870d-08712e64e26a/volumes"
Oct 30 19:18:57.333653 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.333618    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"]
Oct 30 19:18:57.334582 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.334560    4372 cpu_manager.go:414] "containerMap: removing container" podUID="74d7f977-902a-48ea-870d-08712e64e26a" containerName="payload-processing"
Oct 30 19:18:57.334645 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.334583    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d7f977-902a-48ea-870d-08712e64e26a" containerName="payload-processing"
Oct 30 19:18:57.334722 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.334712    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d7f977-902a-48ea-870d-08712e64e26a" containerName="payload-processing"
Oct 30 19:18:57.339667 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.339647    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:18:57.345998 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.345974    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-de4bda\"/\"agc-cred-fcca11cc133a\""
Oct 30 19:18:57.351530 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.351504    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"]
Oct 30 19:18:57.423545 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.423514    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:18:57.423672 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.423589    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:18:57.423758 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.423738    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:18:57.423813 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.423800    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp5qz\" (UniqueName: \"kubernetes.io/projected/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-kube-api-access-rp5qz\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:18:57.524547 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.524517    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:18:57.524667 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.524574    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp5qz\" (UniqueName: \"kubernetes.io/projected/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-kube-api-access-rp5qz\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:18:57.524667 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.524643    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:18:57.524746 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.524690    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:18:57.525327 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.525297    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:18:57.525429 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.525361    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:18:57.527508 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.527488    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:18:57.586245 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.584433    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp5qz\" (UniqueName: \"kubernetes.io/projected/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-kube-api-access-rp5qz\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:18:57.655748 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.655707    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:18:57.703272 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:18:57.703234    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:18:57.951034 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:18:57.951003    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f92f1c_2527_48f7_9271_5fea0d8d4a0e.slice/crio-f20055e956bd9d1d3cb7696d2e6f16814deb985ea65b78fdcc5f53d348973641 WatchSource:0}: Error finding container f20055e956bd9d1d3cb7696d2e6f16814deb985ea65b78fdcc5f53d348973641: Status 404 returned error can't find the container with id f20055e956bd9d1d3cb7696d2e6f16814deb985ea65b78fdcc5f53d348973641
Oct 30 19:18:57.951796 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:57.951768    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"]
Oct 30 19:18:58.059619 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:58.059589    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr" event={"ID":"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e","Type":"ContainerStarted","Data":"5add3f603daefec01ade9bfbe0142c5ee852c6820c244efc45fbd6503a202086"}
Oct 30 19:18:58.059739 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:58.059647    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr" event={"ID":"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e","Type":"ContainerStarted","Data":"f20055e956bd9d1d3cb7696d2e6f16814deb985ea65b78fdcc5f53d348973641"}
Oct 30 19:18:58.059739 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:58.059682    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:18:58.085913 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:58.085863    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr" podStartSLOduration=1.0858501249999999 podStartE2EDuration="1.085850125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:58.081511034 +0000 UTC m=+2751.949929388" watchObservedRunningTime="2026-10-30 19:18:58.085850125 +0000 UTC m=+2751.954268454"
Oct 30 19:18:58.793485 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:58.793454    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"]
Oct 30 19:18:58.796283 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:58.796262    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:18:59.727693 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:59.727657    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"]
Oct 30 19:18:59.734244 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:18:59.734210    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"]
Oct 30 19:19:03.308242 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.308204    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"]
Oct 30 19:19:03.308686 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.308405    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw" podUID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerName="istio-proxy" containerID="cri-o://ee2173f5e31d96f58981ecf5ec7938a27075928702d0fec5fd29404cfee18ea7" gracePeriod=30
Oct 30 19:19:03.691470 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.691424    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"]
Oct 30 19:19:03.696427 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.696406    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:03.702806 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.702780    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-acfdea0a628c\""
Oct 30 19:19:03.702939 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.702922    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-7nkqj\""
Oct 30 19:19:03.708175 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.708149    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"]
Oct 30 19:19:03.779819 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.779787    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3899b6f3-021d-493c-809e-7ba72fcc3d66-routing-overlay-volume\") pod \"payload-processing-external-model-79b45db86b-5d4ls\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:03.779984 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.779960    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3899b6f3-021d-493c-809e-7ba72fcc3d66-plugins-config-volume\") pod \"payload-processing-external-model-79b45db86b-5d4ls\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:03.780038 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.780025    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3899b6f3-021d-493c-809e-7ba72fcc3d66-provider-credentials\") pod \"payload-processing-external-model-79b45db86b-5d4ls\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:03.780308 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.780286    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzthd\" (UniqueName: \"kubernetes.io/projected/3899b6f3-021d-493c-809e-7ba72fcc3d66-kube-api-access-wzthd\") pod \"payload-processing-external-model-79b45db86b-5d4ls\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:03.794202 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.794168    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2" podUID="8e4dbe81-f519-4420-9611-987f44aafd42" containerName="payload-processing" containerID="cri-o://baeff3c5bd324732ae4b4a9605da9c59cc6883f405117d4831451851110a78de" gracePeriod=25
Oct 30 19:19:03.881667 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.881635    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzthd\" (UniqueName: \"kubernetes.io/projected/3899b6f3-021d-493c-809e-7ba72fcc3d66-kube-api-access-wzthd\") pod \"payload-processing-external-model-79b45db86b-5d4ls\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:03.881799 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.881755    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3899b6f3-021d-493c-809e-7ba72fcc3d66-routing-overlay-volume\") pod \"payload-processing-external-model-79b45db86b-5d4ls\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:03.881891 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.881874    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3899b6f3-021d-493c-809e-7ba72fcc3d66-plugins-config-volume\") pod \"payload-processing-external-model-79b45db86b-5d4ls\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:03.881962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.881946    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3899b6f3-021d-493c-809e-7ba72fcc3d66-provider-credentials\") pod \"payload-processing-external-model-79b45db86b-5d4ls\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:03.882450 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.882427    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3899b6f3-021d-493c-809e-7ba72fcc3d66-routing-overlay-volume\") pod \"payload-processing-external-model-79b45db86b-5d4ls\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:03.882571 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.882551    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3899b6f3-021d-493c-809e-7ba72fcc3d66-plugins-config-volume\") pod \"payload-processing-external-model-79b45db86b-5d4ls\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:03.884585 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.884565    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3899b6f3-021d-493c-809e-7ba72fcc3d66-provider-credentials\") pod \"payload-processing-external-model-79b45db86b-5d4ls\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:03.898487 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:03.898466    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzthd\" (UniqueName: \"kubernetes.io/projected/3899b6f3-021d-493c-809e-7ba72fcc3d66-kube-api-access-wzthd\") pod \"payload-processing-external-model-79b45db86b-5d4ls\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:04.015920 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.015887    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:04.561648 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.561332    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:19:04.561648 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.561598    4372 generic.go:356] "Generic (PLEG): container finished" podID="8e4dbe81-f519-4420-9611-987f44aafd42" containerID="baeff3c5bd324732ae4b4a9605da9c59cc6883f405117d4831451851110a78de" exitCode=0
Oct 30 19:19:04.562261 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.561638    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2" event={"ID":"8e4dbe81-f519-4420-9611-987f44aafd42","Type":"ContainerDied","Data":"baeff3c5bd324732ae4b4a9605da9c59cc6883f405117d4831451851110a78de"}
Oct 30 19:19:04.562261 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.561730    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2" event={"ID":"8e4dbe81-f519-4420-9611-987f44aafd42","Type":"ContainerDied","Data":"82b1a9b71de2d391aca177f0adbf93d92ee3e77e253f3b36c84f8ed4fae6b05a"}
Oct 30 19:19:04.562261 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.561757    4372 scope.go:122] "RemoveContainer" containerID="baeff3c5bd324732ae4b4a9605da9c59cc6883f405117d4831451851110a78de"
Oct 30 19:19:04.577279 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.577255    4372 scope.go:122] "RemoveContainer" containerID="baeff3c5bd324732ae4b4a9605da9c59cc6883f405117d4831451851110a78de"
Oct 30 19:19:04.577629 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:19:04.577604    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baeff3c5bd324732ae4b4a9605da9c59cc6883f405117d4831451851110a78de\": container with ID starting with baeff3c5bd324732ae4b4a9605da9c59cc6883f405117d4831451851110a78de not found: ID does not exist" containerID="baeff3c5bd324732ae4b4a9605da9c59cc6883f405117d4831451851110a78de"
Oct 30 19:19:04.577731 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.577644    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baeff3c5bd324732ae4b4a9605da9c59cc6883f405117d4831451851110a78de"} err="failed to get container status \"baeff3c5bd324732ae4b4a9605da9c59cc6883f405117d4831451851110a78de\": rpc error: code = NotFound desc = could not find container \"baeff3c5bd324732ae4b4a9605da9c59cc6883f405117d4831451851110a78de\": container with ID starting with baeff3c5bd324732ae4b4a9605da9c59cc6883f405117d4831451851110a78de not found: ID does not exist"
Oct 30 19:19:04.655006 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.654975    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4dbe81-f519-4420-9611-987f44aafd42-kube-api-access-tr4sn\" (UniqueName: \"kubernetes.io/projected/8e4dbe81-f519-4420-9611-987f44aafd42-kube-api-access-tr4sn\") pod \"8e4dbe81-f519-4420-9611-987f44aafd42\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") "
Oct 30 19:19:04.655278 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.655257    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4dbe81-f519-4420-9611-987f44aafd42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4dbe81-f519-4420-9611-987f44aafd42-plugins-config-volume\") pod \"8e4dbe81-f519-4420-9611-987f44aafd42\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") "
Oct 30 19:19:04.655458 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.655428    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4dbe81-f519-4420-9611-987f44aafd42-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e4dbe81-f519-4420-9611-987f44aafd42-routing-overlay-volume\") pod \"8e4dbe81-f519-4420-9611-987f44aafd42\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") "
Oct 30 19:19:04.655548 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.655516    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4dbe81-f519-4420-9611-987f44aafd42-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e4dbe81-f519-4420-9611-987f44aafd42-provider-credentials\") pod \"8e4dbe81-f519-4420-9611-987f44aafd42\" (UID: \"8e4dbe81-f519-4420-9611-987f44aafd42\") "
Oct 30 19:19:04.655808 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.655733    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4dbe81-f519-4420-9611-987f44aafd42-plugins-config-volume" pod "8e4dbe81-f519-4420-9611-987f44aafd42" (UID: "8e4dbe81-f519-4420-9611-987f44aafd42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:04.655808 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.655817    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4dbe81-f519-4420-9611-987f44aafd42-routing-overlay-volume" pod "8e4dbe81-f519-4420-9611-987f44aafd42" (UID: "8e4dbe81-f519-4420-9611-987f44aafd42"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:04.656201 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.656183    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4dbe81-f519-4420-9611-987f44aafd42-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:04.656288 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.656202    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e4dbe81-f519-4420-9611-987f44aafd42-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:04.657609 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.657586    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4dbe81-f519-4420-9611-987f44aafd42-kube-api-access-tr4sn" pod "8e4dbe81-f519-4420-9611-987f44aafd42" (UID: "8e4dbe81-f519-4420-9611-987f44aafd42"). InnerVolumeSpecName "kube-api-access-tr4sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:04.657742 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.657728    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4dbe81-f519-4420-9611-987f44aafd42-provider-credentials" pod "8e4dbe81-f519-4420-9611-987f44aafd42" (UID: "8e4dbe81-f519-4420-9611-987f44aafd42"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:04.708388 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.708362    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"]
Oct 30 19:19:04.729506 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.729477    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx" podUID="8eb9bec3-a8fd-4795-be5b-43f346c31cb9" containerName="payload-processing" containerID="cri-o://41e0dac994798ed1c6ecb22205cbca697b3dc022378ab3285361d44b65e98b2f" gracePeriod=25
Oct 30 19:19:04.735289 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.735259    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8" podUID="c952c12c-9e03-47a0-b24b-a08f75963f2d" containerName="payload-pre-processing" containerID="cri-o://b241159436f60ea2b4701c265e7069efe7e3378df7f5fe193d950a43167b4449" gracePeriod=25
Oct 30 19:19:04.757972 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.757949    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e4dbe81-f519-4420-9611-987f44aafd42-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:04.758067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:04.757975    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr4sn\" (UniqueName: \"kubernetes.io/projected/8e4dbe81-f519-4420-9611-987f44aafd42-kube-api-access-tr4sn\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:05.571222 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:05.571177    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls" event={"ID":"3899b6f3-021d-493c-809e-7ba72fcc3d66","Type":"ContainerStarted","Data":"55943518450f6f688b0f776443b14efc766827d7409668f1d1070e16bdd278eb"}
Oct 30 19:19:05.571740 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:05.571240    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls" event={"ID":"3899b6f3-021d-493c-809e-7ba72fcc3d66","Type":"ContainerStarted","Data":"cea581314f105ac031923a1543d5ef21efc8808d59d3ddeb72bb9279cd450a25"}
Oct 30 19:19:05.571740 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:05.571275    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:05.572452 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:05.572433    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"
Oct 30 19:19:05.598564 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:05.598509    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls" podStartSLOduration=2.598497667 podStartE2EDuration="2.598497667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:05.59524993 +0000 UTC m=+2759.463668260" watchObservedRunningTime="2026-10-30 19:19:05.598497667 +0000 UTC m=+2759.466915998"
Oct 30 19:19:05.620189 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:05.620163    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"]
Oct 30 19:19:05.626611 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:05.626591    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-865f6676c9-n76w2"]
Oct 30 19:19:05.888843 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:05.888809    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"]
Oct 30 19:19:05.891315 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:05.891293    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:19:06.268265 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:06.268197    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4dbe81-f519-4420-9611-987f44aafd42" path="/var/lib/kubelet/pods/8e4dbe81-f519-4420-9611-987f44aafd42/volumes"
Oct 30 19:19:07.195899 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:07.195866    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"]
Oct 30 19:19:08.400193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:08.375237    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8" podUID="c952c12c-9e03-47a0-b24b-a08f75963f2d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:19:08.813151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:08.812093    4372 generic.go:356] "Generic (PLEG): container finished" podID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerID="ee2173f5e31d96f58981ecf5ec7938a27075928702d0fec5fd29404cfee18ea7" exitCode=0
Oct 30 19:19:08.813151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:08.812272    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw" event={"ID":"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b","Type":"ContainerDied","Data":"ee2173f5e31d96f58981ecf5ec7938a27075928702d0fec5fd29404cfee18ea7"}
Oct 30 19:19:08.813151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:08.812331    4372 scope.go:122] "RemoveContainer" containerID="cc880ea57511314e5161ed64b11449474d54fe1e325a8c25bd0e81a3f734230c"
Oct 30 19:19:08.982090 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:08.981946    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 19:19:09.002192 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.002108    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-data\") pod \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") "
Oct 30 19:19:09.002523 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.002492    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-workload-certs\") pod \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") "
Oct 30 19:19:09.002665 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.002590    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istiod-ca-cert\") pod \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") "
Oct 30 19:19:09.002798 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.002780    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-credential-socket\") pod \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") "
Oct 30 19:19:09.002888 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.002873    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-envoy\") pod \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") "
Oct 30 19:19:09.002939 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.002909    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-workload-socket\") pod \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") "
Oct 30 19:19:09.003046 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.003030    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-kube-api-access-8dwbm\" (UniqueName: \"kubernetes.io/projected/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-kube-api-access-8dwbm\") pod \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") "
Oct 30 19:19:09.003092 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.003060    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-token\" (UniqueName: \"kubernetes.io/projected/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-token\") pod \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") "
Oct 30 19:19:09.003179 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.003105    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-podinfo\") pod \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\" (UID: \"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b\") "
Oct 30 19:19:09.004492 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.002380    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-data" pod "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" (UID: "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:09.004594 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.004466    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-credential-socket" pod "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" (UID: "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:09.004693 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.004672    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-workload-certs" pod "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" (UID: "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:09.004930 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.004912    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istiod-ca-cert" pod "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" (UID: "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:09.005939 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.005893    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-workload-socket" pod "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" (UID: "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:09.007948 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.007925    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-envoy" pod "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" (UID: "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:09.008174 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.008109    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-kube-api-access-8dwbm" pod "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" (UID: "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b"). InnerVolumeSpecName "kube-api-access-8dwbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:09.008288 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.008198    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-podinfo" pod "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" (UID: "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:19:09.009521 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.009493    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-token" pod "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" (UID: "a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:09.105955 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.105930    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dwbm\" (UniqueName: \"kubernetes.io/projected/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-kube-api-access-8dwbm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:09.105955 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.105955    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:09.106154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.105966    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:09.106154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.105975    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:09.106154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.105983    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:09.106154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.105992    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:09.106154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.106002    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:09.106154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.106011    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:09.106154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.106019    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:09.433923 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.433574    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx" podUID="8eb9bec3-a8fd-4795-be5b-43f346c31cb9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:19:09.500522 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.500486    4372 scope.go:122] "RemoveContainer" containerID="ee2173f5e31d96f58981ecf5ec7938a27075928702d0fec5fd29404cfee18ea7"
Oct 30 19:19:09.819846 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.819766    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw" event={"ID":"a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b","Type":"ContainerDied","Data":"7bad246c063b7f1717d7df11125c5f4c3eb8734ba37b1c54b4e818c7180c3385"}
Oct 30 19:19:09.819846 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.819797    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"
Oct 30 19:19:09.868728 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.868690    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"]
Oct 30 19:19:09.872669 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:09.872641    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ssvpw"]
Oct 30 19:19:10.267577 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:10.267545    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" path="/var/lib/kubelet/pods/a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b/volumes"
Oct 30 19:19:10.889108 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:10.889057    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr" podUID="d9f92f1c-2527-48f7-9271-5fea0d8d4a0e" containerName="payload-processing" containerID="cri-o://5add3f603daefec01ade9bfbe0142c5ee852c6820c244efc45fbd6503a202086" gracePeriod=25
Oct 30 19:19:11.186002 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.185964    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:19:11.255146 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.254689    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-provider-credentials\") pod \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") "
Oct 30 19:19:11.255146 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.254939    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-kube-api-access-rp5qz\" (UniqueName: \"kubernetes.io/projected/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-kube-api-access-rp5qz\") pod \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") "
Oct 30 19:19:11.255146 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.255018    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-routing-overlay-volume\") pod \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") "
Oct 30 19:19:11.255146 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.255055    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-plugins-config-volume\") pod \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\" (UID: \"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e\") "
Oct 30 19:19:11.262750 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.258576    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-routing-overlay-volume" pod "d9f92f1c-2527-48f7-9271-5fea0d8d4a0e" (UID: "d9f92f1c-2527-48f7-9271-5fea0d8d4a0e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:11.267785 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.267730    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-plugins-config-volume" pod "d9f92f1c-2527-48f7-9271-5fea0d8d4a0e" (UID: "d9f92f1c-2527-48f7-9271-5fea0d8d4a0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:11.301236 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.301166    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-provider-credentials" pod "d9f92f1c-2527-48f7-9271-5fea0d8d4a0e" (UID: "d9f92f1c-2527-48f7-9271-5fea0d8d4a0e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:11.315244 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.312446    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.315244 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.312481    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.315244 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.312497    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.318732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.318683    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-kube-api-access-rp5qz" pod "d9f92f1c-2527-48f7-9271-5fea0d8d4a0e" (UID: "d9f92f1c-2527-48f7-9271-5fea0d8d4a0e"). InnerVolumeSpecName "kube-api-access-rp5qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:11.414442 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.414363    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp5qz\" (UniqueName: \"kubernetes.io/projected/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e-kube-api-access-rp5qz\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:11.844408 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.844369    4372 generic.go:356] "Generic (PLEG): container finished" podID="d9f92f1c-2527-48f7-9271-5fea0d8d4a0e" containerID="5add3f603daefec01ade9bfbe0142c5ee852c6820c244efc45fbd6503a202086" exitCode=0
Oct 30 19:19:11.844576 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.844438    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr" event={"ID":"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e","Type":"ContainerDied","Data":"5add3f603daefec01ade9bfbe0142c5ee852c6820c244efc45fbd6503a202086"}
Oct 30 19:19:11.844576 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.844445    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"
Oct 30 19:19:11.844576 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.844529    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr" event={"ID":"d9f92f1c-2527-48f7-9271-5fea0d8d4a0e","Type":"ContainerDied","Data":"f20055e956bd9d1d3cb7696d2e6f16814deb985ea65b78fdcc5f53d348973641"}
Oct 30 19:19:11.844576 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.844566    4372 scope.go:122] "RemoveContainer" containerID="5add3f603daefec01ade9bfbe0142c5ee852c6820c244efc45fbd6503a202086"
Oct 30 19:19:11.857878 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.857850    4372 scope.go:122] "RemoveContainer" containerID="5add3f603daefec01ade9bfbe0142c5ee852c6820c244efc45fbd6503a202086"
Oct 30 19:19:11.858220 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:19:11.858198    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5add3f603daefec01ade9bfbe0142c5ee852c6820c244efc45fbd6503a202086\": container with ID starting with 5add3f603daefec01ade9bfbe0142c5ee852c6820c244efc45fbd6503a202086 not found: ID does not exist" containerID="5add3f603daefec01ade9bfbe0142c5ee852c6820c244efc45fbd6503a202086"
Oct 30 19:19:11.858336 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.858227    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5add3f603daefec01ade9bfbe0142c5ee852c6820c244efc45fbd6503a202086"} err="failed to get container status \"5add3f603daefec01ade9bfbe0142c5ee852c6820c244efc45fbd6503a202086\": rpc error: code = NotFound desc = could not find container \"5add3f603daefec01ade9bfbe0142c5ee852c6820c244efc45fbd6503a202086\": container with ID starting with 5add3f603daefec01ade9bfbe0142c5ee852c6820c244efc45fbd6503a202086 not found: ID does not exist"
Oct 30 19:19:11.878675 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.878654    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"]
Oct 30 19:19:11.908668 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:11.908645    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-8684cmwvr"]
Oct 30 19:19:12.267400 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:12.267331    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f92f1c-2527-48f7-9271-5fea0d8d4a0e" path="/var/lib/kubelet/pods/d9f92f1c-2527-48f7-9271-5fea0d8d4a0e/volumes"
Oct 30 19:19:12.641611 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:12.641572    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls" podUID="3899b6f3-021d-493c-809e-7ba72fcc3d66" containerName="payload-processing" containerID="cri-o://55943518450f6f688b0f776443b14efc766827d7409668f1d1070e16bdd278eb" gracePeriod=25
Oct 30 19:19:12.858955 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:12.858925    4372 generic.go:356] "Generic (PLEG): container finished" podID="3899b6f3-021d-493c-809e-7ba72fcc3d66" containerID="55943518450f6f688b0f776443b14efc766827d7409668f1d1070e16bdd278eb" exitCode=0
Oct 30 19:19:12.859087 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:12.859008    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls" event={"ID":"3899b6f3-021d-493c-809e-7ba72fcc3d66","Type":"ContainerDied","Data":"55943518450f6f688b0f776443b14efc766827d7409668f1d1070e16bdd278eb"}
Oct 30 19:19:12.947773 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:12.947747    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:13.040978 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.040946    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3899b6f3-021d-493c-809e-7ba72fcc3d66-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3899b6f3-021d-493c-809e-7ba72fcc3d66-provider-credentials\") pod \"3899b6f3-021d-493c-809e-7ba72fcc3d66\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") "
Oct 30 19:19:13.041215 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.041196    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3899b6f3-021d-493c-809e-7ba72fcc3d66-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3899b6f3-021d-493c-809e-7ba72fcc3d66-routing-overlay-volume\") pod \"3899b6f3-021d-493c-809e-7ba72fcc3d66\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") "
Oct 30 19:19:13.041333 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.041234    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3899b6f3-021d-493c-809e-7ba72fcc3d66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3899b6f3-021d-493c-809e-7ba72fcc3d66-plugins-config-volume\") pod \"3899b6f3-021d-493c-809e-7ba72fcc3d66\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") "
Oct 30 19:19:13.041333 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.041281    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3899b6f3-021d-493c-809e-7ba72fcc3d66-kube-api-access-wzthd\" (UniqueName: \"kubernetes.io/projected/3899b6f3-021d-493c-809e-7ba72fcc3d66-kube-api-access-wzthd\") pod \"3899b6f3-021d-493c-809e-7ba72fcc3d66\" (UID: \"3899b6f3-021d-493c-809e-7ba72fcc3d66\") "
Oct 30 19:19:13.041603 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.041567    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3899b6f3-021d-493c-809e-7ba72fcc3d66-routing-overlay-volume" pod "3899b6f3-021d-493c-809e-7ba72fcc3d66" (UID: "3899b6f3-021d-493c-809e-7ba72fcc3d66"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:13.041688 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.041629    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3899b6f3-021d-493c-809e-7ba72fcc3d66-plugins-config-volume" pod "3899b6f3-021d-493c-809e-7ba72fcc3d66" (UID: "3899b6f3-021d-493c-809e-7ba72fcc3d66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:13.041985 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.041966    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3899b6f3-021d-493c-809e-7ba72fcc3d66-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:13.042036 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.041989    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3899b6f3-021d-493c-809e-7ba72fcc3d66-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:13.043585 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.043565    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3899b6f3-021d-493c-809e-7ba72fcc3d66-kube-api-access-wzthd" pod "3899b6f3-021d-493c-809e-7ba72fcc3d66" (UID: "3899b6f3-021d-493c-809e-7ba72fcc3d66"). InnerVolumeSpecName "kube-api-access-wzthd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:13.043957 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.043931    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3899b6f3-021d-493c-809e-7ba72fcc3d66-provider-credentials" pod "3899b6f3-021d-493c-809e-7ba72fcc3d66" (UID: "3899b6f3-021d-493c-809e-7ba72fcc3d66"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:13.143089 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.143051    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzthd\" (UniqueName: \"kubernetes.io/projected/3899b6f3-021d-493c-809e-7ba72fcc3d66-kube-api-access-wzthd\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:13.143089 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.143084    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3899b6f3-021d-493c-809e-7ba72fcc3d66-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:13.869538 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.869504    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"
Oct 30 19:19:13.869538 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.869513    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls" event={"ID":"3899b6f3-021d-493c-809e-7ba72fcc3d66","Type":"ContainerDied","Data":"cea581314f105ac031923a1543d5ef21efc8808d59d3ddeb72bb9279cd450a25"}
Oct 30 19:19:13.869803 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.869579    4372 scope.go:122] "RemoveContainer" containerID="55943518450f6f688b0f776443b14efc766827d7409668f1d1070e16bdd278eb"
Oct 30 19:19:13.898142 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.898101    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"]
Oct 30 19:19:13.902503 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:13.902481    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-79b45db86b-5d4ls"]
Oct 30 19:19:14.268486 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:14.268401    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3899b6f3-021d-493c-809e-7ba72fcc3d66" path="/var/lib/kubelet/pods/3899b6f3-021d-493c-809e-7ba72fcc3d66/volumes"
Oct 30 19:19:18.310233 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:18.310192    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8" podUID="c952c12c-9e03-47a0-b24b-a08f75963f2d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:19:19.415311 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:19.415268    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx" podUID="8eb9bec3-a8fd-4795-be5b-43f346c31cb9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:19:24.961649 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:24.961309    4372 generic.go:356] "Generic (PLEG): container finished" podID="8eb9bec3-a8fd-4795-be5b-43f346c31cb9" containerID="41e0dac994798ed1c6ecb22205cbca697b3dc022378ab3285361d44b65e98b2f" exitCode=0
Oct 30 19:19:24.961649 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:24.961557    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx" event={"ID":"8eb9bec3-a8fd-4795-be5b-43f346c31cb9","Type":"ContainerDied","Data":"41e0dac994798ed1c6ecb22205cbca697b3dc022378ab3285361d44b65e98b2f"}
Oct 30 19:19:24.964428 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:24.964292    4372 generic.go:356] "Generic (PLEG): container finished" podID="c952c12c-9e03-47a0-b24b-a08f75963f2d" containerID="b241159436f60ea2b4701c265e7069efe7e3378df7f5fe193d950a43167b4449" exitCode=0
Oct 30 19:19:24.964428 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:24.964367    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8" event={"ID":"c952c12c-9e03-47a0-b24b-a08f75963f2d","Type":"ContainerDied","Data":"b241159436f60ea2b4701c265e7069efe7e3378df7f5fe193d950a43167b4449"}
Oct 30 19:19:25.043573 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.043547    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"
Oct 30 19:19:25.077614 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.077582    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"
Oct 30 19:19:25.118835 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.118796    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c952c12c-9e03-47a0-b24b-a08f75963f2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c952c12c-9e03-47a0-b24b-a08f75963f2d-plugins-config-volume\") pod \"c952c12c-9e03-47a0-b24b-a08f75963f2d\" (UID: \"c952c12c-9e03-47a0-b24b-a08f75963f2d\") "
Oct 30 19:19:25.118982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.118866    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb9bec3-a8fd-4795-be5b-43f346c31cb9-kube-api-access-crdmn\" (UniqueName: \"kubernetes.io/projected/8eb9bec3-a8fd-4795-be5b-43f346c31cb9-kube-api-access-crdmn\") pod \"8eb9bec3-a8fd-4795-be5b-43f346c31cb9\" (UID: \"8eb9bec3-a8fd-4795-be5b-43f346c31cb9\") "
Oct 30 19:19:25.118982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.118896    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb9bec3-a8fd-4795-be5b-43f346c31cb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb9bec3-a8fd-4795-be5b-43f346c31cb9-plugins-config-volume\") pod \"8eb9bec3-a8fd-4795-be5b-43f346c31cb9\" (UID: \"8eb9bec3-a8fd-4795-be5b-43f346c31cb9\") "
Oct 30 19:19:25.119111 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.118986    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c952c12c-9e03-47a0-b24b-a08f75963f2d-kube-api-access-l85zb\" (UniqueName: \"kubernetes.io/projected/c952c12c-9e03-47a0-b24b-a08f75963f2d-kube-api-access-l85zb\") pod \"c952c12c-9e03-47a0-b24b-a08f75963f2d\" (UID: \"c952c12c-9e03-47a0-b24b-a08f75963f2d\") "
Oct 30 19:19:25.119335 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.119304    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c952c12c-9e03-47a0-b24b-a08f75963f2d-plugins-config-volume" pod "c952c12c-9e03-47a0-b24b-a08f75963f2d" (UID: "c952c12c-9e03-47a0-b24b-a08f75963f2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:25.119424 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.119375    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb9bec3-a8fd-4795-be5b-43f346c31cb9-plugins-config-volume" pod "8eb9bec3-a8fd-4795-be5b-43f346c31cb9" (UID: "8eb9bec3-a8fd-4795-be5b-43f346c31cb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:25.119780 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.119760    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c952c12c-9e03-47a0-b24b-a08f75963f2d-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:25.119861 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.119784    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb9bec3-a8fd-4795-be5b-43f346c31cb9-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:25.121259 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.121237    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb9bec3-a8fd-4795-be5b-43f346c31cb9-kube-api-access-crdmn" pod "8eb9bec3-a8fd-4795-be5b-43f346c31cb9" (UID: "8eb9bec3-a8fd-4795-be5b-43f346c31cb9"). InnerVolumeSpecName "kube-api-access-crdmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:25.121381 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.121361    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c952c12c-9e03-47a0-b24b-a08f75963f2d-kube-api-access-l85zb" pod "c952c12c-9e03-47a0-b24b-a08f75963f2d" (UID: "c952c12c-9e03-47a0-b24b-a08f75963f2d"). InnerVolumeSpecName "kube-api-access-l85zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:25.225747 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.225712    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crdmn\" (UniqueName: \"kubernetes.io/projected/8eb9bec3-a8fd-4795-be5b-43f346c31cb9-kube-api-access-crdmn\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:25.225747 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.225742    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l85zb\" (UniqueName: \"kubernetes.io/projected/c952c12c-9e03-47a0-b24b-a08f75963f2d-kube-api-access-l85zb\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:25.973392 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.973363    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"
Oct 30 19:19:25.973888 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.973390    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8" event={"ID":"c952c12c-9e03-47a0-b24b-a08f75963f2d","Type":"ContainerDied","Data":"e5e2b99ff85f3ec48ae7a4ae1eb4c1eab217b008b1a88919da93238a19c1ebf8"}
Oct 30 19:19:25.973888 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.973454    4372 scope.go:122] "RemoveContainer" containerID="b241159436f60ea2b4701c265e7069efe7e3378df7f5fe193d950a43167b4449"
Oct 30 19:19:25.975054 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.975014    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx" event={"ID":"8eb9bec3-a8fd-4795-be5b-43f346c31cb9","Type":"ContainerDied","Data":"f5f3e3d9c45103a15db3193e76972845470a65b23c649ce1241fb3a2e08d9797"}
Oct 30 19:19:25.975054 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.975047    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"
Oct 30 19:19:25.990003 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:25.989988    4372 scope.go:122] "RemoveContainer" containerID="41e0dac994798ed1c6ecb22205cbca697b3dc022378ab3285361d44b65e98b2f"
Oct 30 19:19:26.007726 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.007672    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"]
Oct 30 19:19:26.012192 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.012171    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-689c23-76688884f9-k2vp8"]
Oct 30 19:19:26.024528 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.024510    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"]
Oct 30 19:19:26.033896 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.033874    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-689c23-74b4f84f9d-wwwxx"]
Oct 30 19:19:26.268056 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.267977    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb9bec3-a8fd-4795-be5b-43f346c31cb9" path="/var/lib/kubelet/pods/8eb9bec3-a8fd-4795-be5b-43f346c31cb9/volumes"
Oct 30 19:19:26.268474 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.268456    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c952c12c-9e03-47a0-b24b-a08f75963f2d" path="/var/lib/kubelet/pods/c952c12c-9e03-47a0-b24b-a08f75963f2d/volumes"
Oct 30 19:19:26.797461 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.797432    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"]
Oct 30 19:19:26.798218 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798201    4372 cpu_manager.go:414] "containerMap: removing container" podUID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerName="istio-proxy"
Oct 30 19:19:26.798290 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798219    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerName="istio-proxy"
Oct 30 19:19:26.798290 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798233    4372 cpu_manager.go:414] "containerMap: removing container" podUID="8e4dbe81-f519-4420-9611-987f44aafd42" containerName="payload-processing"
Oct 30 19:19:26.798290 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798238    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4dbe81-f519-4420-9611-987f44aafd42" containerName="payload-processing"
Oct 30 19:19:26.798290 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798246    4372 cpu_manager.go:414] "containerMap: removing container" podUID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerName="istio-proxy"
Oct 30 19:19:26.798290 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798251    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerName="istio-proxy"
Oct 30 19:19:26.798290 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798263    4372 cpu_manager.go:414] "containerMap: removing container" podUID="3899b6f3-021d-493c-809e-7ba72fcc3d66" containerName="payload-processing"
Oct 30 19:19:26.798290 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798268    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3899b6f3-021d-493c-809e-7ba72fcc3d66" containerName="payload-processing"
Oct 30 19:19:26.798290 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798279    4372 cpu_manager.go:414] "containerMap: removing container" podUID="8eb9bec3-a8fd-4795-be5b-43f346c31cb9" containerName="payload-processing"
Oct 30 19:19:26.798290 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798284    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb9bec3-a8fd-4795-be5b-43f346c31cb9" containerName="payload-processing"
Oct 30 19:19:26.798581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798305    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d9f92f1c-2527-48f7-9271-5fea0d8d4a0e" containerName="payload-processing"
Oct 30 19:19:26.798581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798310    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f92f1c-2527-48f7-9271-5fea0d8d4a0e" containerName="payload-processing"
Oct 30 19:19:26.798581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798334    4372 cpu_manager.go:414] "containerMap: removing container" podUID="c952c12c-9e03-47a0-b24b-a08f75963f2d" containerName="payload-pre-processing"
Oct 30 19:19:26.798581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798338    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c952c12c-9e03-47a0-b24b-a08f75963f2d" containerName="payload-pre-processing"
Oct 30 19:19:26.798581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798446    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="c952c12c-9e03-47a0-b24b-a08f75963f2d" containerName="payload-pre-processing"
Oct 30 19:19:26.798581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798457    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerName="istio-proxy"
Oct 30 19:19:26.798581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798466    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4dbe81-f519-4420-9611-987f44aafd42" containerName="payload-processing"
Oct 30 19:19:26.798581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798472    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a5e1e6-da62-4ccc-853e-d7346ac1ad5b" containerName="istio-proxy"
Oct 30 19:19:26.798581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798479    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="3899b6f3-021d-493c-809e-7ba72fcc3d66" containerName="payload-processing"
Oct 30 19:19:26.798581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798485    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb9bec3-a8fd-4795-be5b-43f346c31cb9" containerName="payload-processing"
Oct 30 19:19:26.798581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.798493    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f92f1c-2527-48f7-9271-5fea0d8d4a0e" containerName="payload-processing"
Oct 30 19:19:26.803552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.803532    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:26.813606 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.813587    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"]
Oct 30 19:19:26.850929 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.850895    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abe5746-4718-4d28-88dc-5b9a54999869-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:26.851403 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.851375    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6abe5746-4718-4d28-88dc-5b9a54999869-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:26.851597 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.851566    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls8jc\" (UniqueName: \"kubernetes.io/projected/6abe5746-4718-4d28-88dc-5b9a54999869-kube-api-access-ls8jc\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:26.851762 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:26.851741    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6abe5746-4718-4d28-88dc-5b9a54999869-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:27.065528 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.010461    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abe5746-4718-4d28-88dc-5b9a54999869-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:27.076422 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.076387    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6abe5746-4718-4d28-88dc-5b9a54999869-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:27.076748 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.076724    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls8jc\" (UniqueName: \"kubernetes.io/projected/6abe5746-4718-4d28-88dc-5b9a54999869-kube-api-access-ls8jc\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:27.076932 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.053887    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abe5746-4718-4d28-88dc-5b9a54999869-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:27.077697 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.077163    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6abe5746-4718-4d28-88dc-5b9a54999869-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:27.080015 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.078643    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6abe5746-4718-4d28-88dc-5b9a54999869-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:27.132169 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.125768    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6abe5746-4718-4d28-88dc-5b9a54999869-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:27.155481 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.153501    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls8jc\" (UniqueName: \"kubernetes.io/projected/6abe5746-4718-4d28-88dc-5b9a54999869-kube-api-access-ls8jc\") pod \"payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") " pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:27.440152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.439421    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:27.473867 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.470183    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"]
Oct 30 19:19:27.486506 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.486480    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:27.522631 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.514596    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"]
Oct 30 19:19:27.619163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.614678    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:27.619163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.614835    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:27.619163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.614902    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:27.619163 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.614999    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr8hw\" (UniqueName: \"kubernetes.io/projected/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-kube-api-access-vr8hw\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:27.763221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.748156    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:27.763221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.748331    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:27.763221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.748403    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:27.763221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.748522    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr8hw\" (UniqueName: \"kubernetes.io/projected/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-kube-api-access-vr8hw\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:27.763221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.749867    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:27.764196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.764150    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:27.800386 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.799470    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:27.810150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.805482    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr8hw\" (UniqueName: \"kubernetes.io/projected/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-kube-api-access-vr8hw\") pod \"payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") " pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:27.866166 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:27.865548    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:27.872155 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:19:27.871861    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:19:28.125718 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:28.112379    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"]
Oct 30 19:19:28.131760 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:19:28.130616    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abe5746_4718_4d28_88dc_5b9a54999869.slice/crio-252635cefba26fbb94989bbd832917d16c1acf26d37afc868f78f9afd76ec619 WatchSource:0}: Error finding container 252635cefba26fbb94989bbd832917d16c1acf26d37afc868f78f9afd76ec619: Status 404 returned error can't find the container with id 252635cefba26fbb94989bbd832917d16c1acf26d37afc868f78f9afd76ec619
Oct 30 19:19:28.173835 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:28.173803    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8" event={"ID":"6abe5746-4718-4d28-88dc-5b9a54999869","Type":"ContainerStarted","Data":"252635cefba26fbb94989bbd832917d16c1acf26d37afc868f78f9afd76ec619"}
Oct 30 19:19:28.205050 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:19:28.205020    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09ce3e3_0b45_4f34_926f_a6a067ef44a4.slice/crio-9b9fa2531a070cefefde9c9dd9e726b862aadd33cf47b83755e5b346f74b093a WatchSource:0}: Error finding container 9b9fa2531a070cefefde9c9dd9e726b862aadd33cf47b83755e5b346f74b093a: Status 404 returned error can't find the container with id 9b9fa2531a070cefefde9c9dd9e726b862aadd33cf47b83755e5b346f74b093a
Oct 30 19:19:28.206956 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:28.206929    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"]
Oct 30 19:19:29.182810 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:29.182766    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8" event={"ID":"6abe5746-4718-4d28-88dc-5b9a54999869","Type":"ContainerStarted","Data":"9a645b2b558259c44ec3f2d04c256e92656e9eaa042da54d1ebdd05b56dfb02a"}
Oct 30 19:19:29.183280 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:29.182998    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:29.184609 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:29.184582    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6" event={"ID":"e09ce3e3-0b45-4f34-926f-a6a067ef44a4","Type":"ContainerStarted","Data":"d58740b9ee150659db9d8b2d41285d66f9a3955705bbc7d81224a29a7f48f4aa"}
Oct 30 19:19:29.184737 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:29.184635    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6" event={"ID":"e09ce3e3-0b45-4f34-926f-a6a067ef44a4","Type":"ContainerStarted","Data":"9b9fa2531a070cefefde9c9dd9e726b862aadd33cf47b83755e5b346f74b093a"}
Oct 30 19:19:29.184737 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:29.184688    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:29.214049 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:29.214008    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8" podStartSLOduration=3.21399613 podStartE2EDuration="3.21399613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:29.212503493 +0000 UTC m=+2783.080921816" watchObservedRunningTime="2026-10-30 19:19:29.21399613 +0000 UTC m=+2783.082414460"
Oct 30 19:19:29.247458 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:29.247416    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6" podStartSLOduration=2.247403518 podStartE2EDuration="2.247403518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:29.243838563 +0000 UTC m=+2783.112256893" watchObservedRunningTime="2026-10-30 19:19:29.247403518 +0000 UTC m=+2783.115821848"
Oct 30 19:19:29.790184 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:29.785454    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"]
Oct 30 19:19:29.790184 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:29.785750    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb" podUID="fbcd714e-b581-4142-ae56-05bf2a33d52b" containerName="istio-proxy" containerID="cri-o://c16eec50949089e15a1cd73b971848a5576c218980eca67630c86732b0829293" gracePeriod=30
Oct 30 19:19:32.163432 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:32.163396    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:19:32.566867 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:32.566781    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:19:32.566867 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:32.566845    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb" podUID="fbcd714e-b581-4142-ae56-05bf2a33d52b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:19:35.341846 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.341810    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:19:35.360269 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.360241    4372 generic.go:356] "Generic (PLEG): container finished" podID="fbcd714e-b581-4142-ae56-05bf2a33d52b" containerID="c16eec50949089e15a1cd73b971848a5576c218980eca67630c86732b0829293" exitCode=0
Oct 30 19:19:35.360378 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.360279    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"
Oct 30 19:19:35.360378 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.360323    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb" event={"ID":"fbcd714e-b581-4142-ae56-05bf2a33d52b","Type":"ContainerDied","Data":"c16eec50949089e15a1cd73b971848a5576c218980eca67630c86732b0829293"}
Oct 30 19:19:35.360458 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.360378    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb" event={"ID":"fbcd714e-b581-4142-ae56-05bf2a33d52b","Type":"ContainerDied","Data":"392a42698e3b8b80fc421ebb5a9a3264c4b88d7d901d14c2aa7fe202feabe18b"}
Oct 30 19:19:35.360458 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.360411    4372 scope.go:122] "RemoveContainer" containerID="c16eec50949089e15a1cd73b971848a5576c218980eca67630c86732b0829293"
Oct 30 19:19:35.371394 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.371374    4372 scope.go:122] "RemoveContainer" containerID="c16eec50949089e15a1cd73b971848a5576c218980eca67630c86732b0829293"
Oct 30 19:19:35.371748 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:19:35.371722    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c16eec50949089e15a1cd73b971848a5576c218980eca67630c86732b0829293\": container with ID starting with c16eec50949089e15a1cd73b971848a5576c218980eca67630c86732b0829293 not found: ID does not exist" containerID="c16eec50949089e15a1cd73b971848a5576c218980eca67630c86732b0829293"
Oct 30 19:19:35.371847 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.371761    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16eec50949089e15a1cd73b971848a5576c218980eca67630c86732b0829293"} err="failed to get container status \"c16eec50949089e15a1cd73b971848a5576c218980eca67630c86732b0829293\": rpc error: code = NotFound desc = could not find container \"c16eec50949089e15a1cd73b971848a5576c218980eca67630c86732b0829293\": container with ID starting with c16eec50949089e15a1cd73b971848a5576c218980eca67630c86732b0829293 not found: ID does not exist"
Oct 30 19:19:35.558206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.557495    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-token\" (UniqueName: \"kubernetes.io/projected/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-token\") pod \"fbcd714e-b581-4142-ae56-05bf2a33d52b\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") "
Oct 30 19:19:35.558206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.557711    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbcd714e-b581-4142-ae56-05bf2a33d52b-kube-api-access-g2kgf\" (UniqueName: \"kubernetes.io/projected/fbcd714e-b581-4142-ae56-05bf2a33d52b-kube-api-access-g2kgf\") pod \"fbcd714e-b581-4142-ae56-05bf2a33d52b\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") "
Oct 30 19:19:35.558206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.557775    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbcd714e-b581-4142-ae56-05bf2a33d52b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbcd714e-b581-4142-ae56-05bf2a33d52b-istiod-ca-cert\") pod \"fbcd714e-b581-4142-ae56-05bf2a33d52b\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") "
Oct 30 19:19:35.558206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.558148    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-workload-socket\") pod \"fbcd714e-b581-4142-ae56-05bf2a33d52b\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") "
Oct 30 19:19:35.558206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.558206    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-credential-socket\") pod \"fbcd714e-b581-4142-ae56-05bf2a33d52b\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") "
Oct 30 19:19:35.558651 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.558235    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-data\") pod \"fbcd714e-b581-4142-ae56-05bf2a33d52b\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") "
Oct 30 19:19:35.602327 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.558711    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-workload-certs\") pod \"fbcd714e-b581-4142-ae56-05bf2a33d52b\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") "
Oct 30 19:19:35.602327 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.558883    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-podinfo\") pod \"fbcd714e-b581-4142-ae56-05bf2a33d52b\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") "
Oct 30 19:19:35.602327 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.558942    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-envoy\") pod \"fbcd714e-b581-4142-ae56-05bf2a33d52b\" (UID: \"fbcd714e-b581-4142-ae56-05bf2a33d52b\") "
Oct 30 19:19:35.602327 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.572932    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-workload-socket" pod "fbcd714e-b581-4142-ae56-05bf2a33d52b" (UID: "fbcd714e-b581-4142-ae56-05bf2a33d52b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:35.602327 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.573993    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-credential-socket" pod "fbcd714e-b581-4142-ae56-05bf2a33d52b" (UID: "fbcd714e-b581-4142-ae56-05bf2a33d52b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:35.602327 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.574719    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-workload-certs" pod "fbcd714e-b581-4142-ae56-05bf2a33d52b" (UID: "fbcd714e-b581-4142-ae56-05bf2a33d52b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:35.602327 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.575814    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-data" pod "fbcd714e-b581-4142-ae56-05bf2a33d52b" (UID: "fbcd714e-b581-4142-ae56-05bf2a33d52b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:35.602327 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.576271    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbcd714e-b581-4142-ae56-05bf2a33d52b-istiod-ca-cert" pod "fbcd714e-b581-4142-ae56-05bf2a33d52b" (UID: "fbcd714e-b581-4142-ae56-05bf2a33d52b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:35.663185 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.663019    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbcd714e-b581-4142-ae56-05bf2a33d52b-kube-api-access-g2kgf" pod "fbcd714e-b581-4142-ae56-05bf2a33d52b" (UID: "fbcd714e-b581-4142-ae56-05bf2a33d52b"). InnerVolumeSpecName "kube-api-access-g2kgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:35.664393 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.664358    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:35.664529 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.664399    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:35.664529 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.664417    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:35.664529 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.664431    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:35.664529 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.664446    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2kgf\" (UniqueName: \"kubernetes.io/projected/fbcd714e-b581-4142-ae56-05bf2a33d52b-kube-api-access-g2kgf\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:35.664529 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.664462    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbcd714e-b581-4142-ae56-05bf2a33d52b-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:35.666438 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.666334    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-envoy" pod "fbcd714e-b581-4142-ae56-05bf2a33d52b" (UID: "fbcd714e-b581-4142-ae56-05bf2a33d52b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:35.666438 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.666397    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-token" pod "fbcd714e-b581-4142-ae56-05bf2a33d52b" (UID: "fbcd714e-b581-4142-ae56-05bf2a33d52b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:35.666942 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.666913    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-podinfo" pod "fbcd714e-b581-4142-ae56-05bf2a33d52b" (UID: "fbcd714e-b581-4142-ae56-05bf2a33d52b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:19:35.766518 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.766486    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:35.766627 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.766525    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:35.766627 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:35.766541    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbcd714e-b581-4142-ae56-05bf2a33d52b-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:36.056151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:36.055750    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"]
Oct 30 19:19:36.064738 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:36.064706    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-9fjcb"]
Oct 30 19:19:36.271593 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:36.271513    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbcd714e-b581-4142-ae56-05bf2a33d52b" path="/var/lib/kubelet/pods/fbcd714e-b581-4142-ae56-05bf2a33d52b/volumes"
Oct 30 19:19:40.197093 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:40.197063    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:40.197561 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:40.197234    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:40.272322 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:40.272289    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"]
Oct 30 19:19:40.288509 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:40.288475    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"]
Oct 30 19:19:45.273330 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.273285    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d" podUID="9c89796b-9ed3-4d26-a3e1-89525585be35" containerName="payload-processing" containerID="cri-o://8d095bc22255e5c476b762f6e1c4e1b50265d11f2a6b36f82582790001f3192f" gracePeriod=25
Oct 30 19:19:45.289710 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.289679    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc" podUID="6faafbec-abd9-4f16-a979-e607b39565ee" containerName="payload-processing" containerID="cri-o://a843d11a3d6fb2645156537a968d84c4ac9f5be879895aaf2bf0ed0e7e1f68cf" gracePeriod=25
Oct 30 19:19:45.453881 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.453850    4372 generic.go:356] "Generic (PLEG): container finished" podID="6faafbec-abd9-4f16-a979-e607b39565ee" containerID="a843d11a3d6fb2645156537a968d84c4ac9f5be879895aaf2bf0ed0e7e1f68cf" exitCode=0
Oct 30 19:19:45.454082 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.454060    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc" event={"ID":"6faafbec-abd9-4f16-a979-e607b39565ee","Type":"ContainerDied","Data":"a843d11a3d6fb2645156537a968d84c4ac9f5be879895aaf2bf0ed0e7e1f68cf"}
Oct 30 19:19:45.456257 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.456230    4372 generic.go:356] "Generic (PLEG): container finished" podID="9c89796b-9ed3-4d26-a3e1-89525585be35" containerID="8d095bc22255e5c476b762f6e1c4e1b50265d11f2a6b36f82582790001f3192f" exitCode=0
Oct 30 19:19:45.456378 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.456300    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d" event={"ID":"9c89796b-9ed3-4d26-a3e1-89525585be35","Type":"ContainerDied","Data":"8d095bc22255e5c476b762f6e1c4e1b50265d11f2a6b36f82582790001f3192f"}
Oct 30 19:19:45.641820 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.641792    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:19:45.666825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.666791    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:19:45.740088 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.740058    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c89796b-9ed3-4d26-a3e1-89525585be35-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c89796b-9ed3-4d26-a3e1-89525585be35-provider-credentials\") pod \"9c89796b-9ed3-4d26-a3e1-89525585be35\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") "
Oct 30 19:19:45.740301 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.740144    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c89796b-9ed3-4d26-a3e1-89525585be35-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c89796b-9ed3-4d26-a3e1-89525585be35-routing-overlay-volume\") pod \"9c89796b-9ed3-4d26-a3e1-89525585be35\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") "
Oct 30 19:19:45.740301 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.740184    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c89796b-9ed3-4d26-a3e1-89525585be35-kube-api-access-bd4d5\" (UniqueName: \"kubernetes.io/projected/9c89796b-9ed3-4d26-a3e1-89525585be35-kube-api-access-bd4d5\") pod \"9c89796b-9ed3-4d26-a3e1-89525585be35\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") "
Oct 30 19:19:45.740301 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.740213    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6faafbec-abd9-4f16-a979-e607b39565ee-kube-api-access-s8shg\" (UniqueName: \"kubernetes.io/projected/6faafbec-abd9-4f16-a979-e607b39565ee-kube-api-access-s8shg\") pod \"6faafbec-abd9-4f16-a979-e607b39565ee\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") "
Oct 30 19:19:45.740487 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.740309    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c89796b-9ed3-4d26-a3e1-89525585be35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c89796b-9ed3-4d26-a3e1-89525585be35-plugins-config-volume\") pod \"9c89796b-9ed3-4d26-a3e1-89525585be35\" (UID: \"9c89796b-9ed3-4d26-a3e1-89525585be35\") "
Oct 30 19:19:45.740487 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.740336    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6faafbec-abd9-4f16-a979-e607b39565ee-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6faafbec-abd9-4f16-a979-e607b39565ee-provider-credentials\") pod \"6faafbec-abd9-4f16-a979-e607b39565ee\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") "
Oct 30 19:19:45.740487 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.740450    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6faafbec-abd9-4f16-a979-e607b39565ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6faafbec-abd9-4f16-a979-e607b39565ee-plugins-config-volume\") pod \"6faafbec-abd9-4f16-a979-e607b39565ee\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") "
Oct 30 19:19:45.740675 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.740495    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6faafbec-abd9-4f16-a979-e607b39565ee-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6faafbec-abd9-4f16-a979-e607b39565ee-routing-overlay-volume\") pod \"6faafbec-abd9-4f16-a979-e607b39565ee\" (UID: \"6faafbec-abd9-4f16-a979-e607b39565ee\") "
Oct 30 19:19:45.740675 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.740506    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c89796b-9ed3-4d26-a3e1-89525585be35-routing-overlay-volume" pod "9c89796b-9ed3-4d26-a3e1-89525585be35" (UID: "9c89796b-9ed3-4d26-a3e1-89525585be35"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:45.741067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.740977    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6faafbec-abd9-4f16-a979-e607b39565ee-routing-overlay-volume" pod "6faafbec-abd9-4f16-a979-e607b39565ee" (UID: "6faafbec-abd9-4f16-a979-e607b39565ee"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:45.741239 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.741166    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6faafbec-abd9-4f16-a979-e607b39565ee-plugins-config-volume" pod "6faafbec-abd9-4f16-a979-e607b39565ee" (UID: "6faafbec-abd9-4f16-a979-e607b39565ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:45.741425 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.741404    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6faafbec-abd9-4f16-a979-e607b39565ee-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:45.741503 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.741431    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6faafbec-abd9-4f16-a979-e607b39565ee-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:45.741503 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.741446    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c89796b-9ed3-4d26-a3e1-89525585be35-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:45.741656 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.741637    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c89796b-9ed3-4d26-a3e1-89525585be35-plugins-config-volume" pod "9c89796b-9ed3-4d26-a3e1-89525585be35" (UID: "9c89796b-9ed3-4d26-a3e1-89525585be35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:45.742989 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.742967    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c89796b-9ed3-4d26-a3e1-89525585be35-kube-api-access-bd4d5" pod "9c89796b-9ed3-4d26-a3e1-89525585be35" (UID: "9c89796b-9ed3-4d26-a3e1-89525585be35"). InnerVolumeSpecName "kube-api-access-bd4d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:45.743110 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.743055    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c89796b-9ed3-4d26-a3e1-89525585be35-provider-credentials" pod "9c89796b-9ed3-4d26-a3e1-89525585be35" (UID: "9c89796b-9ed3-4d26-a3e1-89525585be35"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:45.743110 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.743084    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6faafbec-abd9-4f16-a979-e607b39565ee-kube-api-access-s8shg" pod "6faafbec-abd9-4f16-a979-e607b39565ee" (UID: "6faafbec-abd9-4f16-a979-e607b39565ee"). InnerVolumeSpecName "kube-api-access-s8shg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:45.743443 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.743425    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6faafbec-abd9-4f16-a979-e607b39565ee-provider-credentials" pod "6faafbec-abd9-4f16-a979-e607b39565ee" (UID: "6faafbec-abd9-4f16-a979-e607b39565ee"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:45.843334 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.843302    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c89796b-9ed3-4d26-a3e1-89525585be35-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:45.843334 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.843333    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd4d5\" (UniqueName: \"kubernetes.io/projected/9c89796b-9ed3-4d26-a3e1-89525585be35-kube-api-access-bd4d5\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:45.843552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.843343    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8shg\" (UniqueName: \"kubernetes.io/projected/6faafbec-abd9-4f16-a979-e607b39565ee-kube-api-access-s8shg\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:45.843552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.843352    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c89796b-9ed3-4d26-a3e1-89525585be35-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:45.843552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:45.843361    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6faafbec-abd9-4f16-a979-e607b39565ee-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:46.467783 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:46.467738    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d" event={"ID":"9c89796b-9ed3-4d26-a3e1-89525585be35","Type":"ContainerDied","Data":"f90f81e2be9ef5162e9bb2cfaca62ab5a36e18294b79105c06ea203c723929e3"}
Oct 30 19:19:46.468402 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:46.467814    4372 scope.go:122] "RemoveContainer" containerID="8d095bc22255e5c476b762f6e1c4e1b50265d11f2a6b36f82582790001f3192f"
Oct 30 19:19:46.468402 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:46.467988    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"
Oct 30 19:19:46.473596 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:46.472443    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc" event={"ID":"6faafbec-abd9-4f16-a979-e607b39565ee","Type":"ContainerDied","Data":"e510104573f672bf5a8135005f5c3a750c65fdfcd0c2e3ee967d841d1161befb"}
Oct 30 19:19:46.473596 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:46.472604    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"
Oct 30 19:19:46.493208 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:46.493184    4372 scope.go:122] "RemoveContainer" containerID="a843d11a3d6fb2645156537a968d84c4ac9f5be879895aaf2bf0ed0e7e1f68cf"
Oct 30 19:19:46.515323 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:46.515290    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"]
Oct 30 19:19:46.523810 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:46.523780    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-5449sldhc"]
Oct 30 19:19:46.550566 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:46.549691    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"]
Oct 30 19:19:46.562231 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:46.562200    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bd4hnj4d"]
Oct 30 19:19:48.266977 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:48.266947    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6faafbec-abd9-4f16-a979-e607b39565ee" path="/var/lib/kubelet/pods/6faafbec-abd9-4f16-a979-e607b39565ee/volumes"
Oct 30 19:19:48.267444 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:48.267427    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c89796b-9ed3-4d26-a3e1-89525585be35" path="/var/lib/kubelet/pods/9c89796b-9ed3-4d26-a3e1-89525585be35/volumes"
Oct 30 19:19:49.604295 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:49.604249    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"]
Oct 30 19:19:50.378241 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:50.378210    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"]
Oct 30 19:19:54.605366 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:54.605325    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8" podUID="6abe5746-4718-4d28-88dc-5b9a54999869" containerName="payload-processing" containerID="cri-o://9a645b2b558259c44ec3f2d04c256e92656e9eaa042da54d1ebdd05b56dfb02a" gracePeriod=25
Oct 30 19:19:54.900665 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:54.900638    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:55.022769 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.022738    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6abe5746-4718-4d28-88dc-5b9a54999869-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abe5746-4718-4d28-88dc-5b9a54999869-plugins-config-volume\") pod \"6abe5746-4718-4d28-88dc-5b9a54999869\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") "
Oct 30 19:19:55.022960 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.022910    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abe5746-4718-4d28-88dc-5b9a54999869-kube-api-access-ls8jc\" (UniqueName: \"kubernetes.io/projected/6abe5746-4718-4d28-88dc-5b9a54999869-kube-api-access-ls8jc\") pod \"6abe5746-4718-4d28-88dc-5b9a54999869\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") "
Oct 30 19:19:55.022960 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.022948    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abe5746-4718-4d28-88dc-5b9a54999869-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6abe5746-4718-4d28-88dc-5b9a54999869-provider-credentials\") pod \"6abe5746-4718-4d28-88dc-5b9a54999869\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") "
Oct 30 19:19:55.023173 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.023112    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6abe5746-4718-4d28-88dc-5b9a54999869-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6abe5746-4718-4d28-88dc-5b9a54999869-routing-overlay-volume\") pod \"6abe5746-4718-4d28-88dc-5b9a54999869\" (UID: \"6abe5746-4718-4d28-88dc-5b9a54999869\") "
Oct 30 19:19:55.023281 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.023251    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abe5746-4718-4d28-88dc-5b9a54999869-plugins-config-volume" pod "6abe5746-4718-4d28-88dc-5b9a54999869" (UID: "6abe5746-4718-4d28-88dc-5b9a54999869"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:55.023507 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.023479    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abe5746-4718-4d28-88dc-5b9a54999869-routing-overlay-volume" pod "6abe5746-4718-4d28-88dc-5b9a54999869" (UID: "6abe5746-4718-4d28-88dc-5b9a54999869"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:55.023830 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.023810    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abe5746-4718-4d28-88dc-5b9a54999869-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:55.023922 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.023833    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6abe5746-4718-4d28-88dc-5b9a54999869-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:55.025429 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.025401    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abe5746-4718-4d28-88dc-5b9a54999869-kube-api-access-ls8jc" pod "6abe5746-4718-4d28-88dc-5b9a54999869" (UID: "6abe5746-4718-4d28-88dc-5b9a54999869"). InnerVolumeSpecName "kube-api-access-ls8jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:55.025519 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.025437    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abe5746-4718-4d28-88dc-5b9a54999869-provider-credentials" pod "6abe5746-4718-4d28-88dc-5b9a54999869" (UID: "6abe5746-4718-4d28-88dc-5b9a54999869"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:55.125674 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.125618    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls8jc\" (UniqueName: \"kubernetes.io/projected/6abe5746-4718-4d28-88dc-5b9a54999869-kube-api-access-ls8jc\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:55.125674 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.125644    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6abe5746-4718-4d28-88dc-5b9a54999869-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:55.276447 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.276415    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"]
Oct 30 19:19:55.276618 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.276599    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerName="istio-proxy" containerID="cri-o://34914257f0fe610ae9da5ecb927e9b5e4e57f23f1acde2a2d9b94e875c1345a1" gracePeriod=30
Oct 30 19:19:55.386206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.385408    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6" podUID="e09ce3e3-0b45-4f34-926f-a6a067ef44a4" containerName="payload-processing" containerID="cri-o://d58740b9ee150659db9d8b2d41285d66f9a3955705bbc7d81224a29a7f48f4aa" gracePeriod=25
Oct 30 19:19:55.591373 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.591334    4372 generic.go:356] "Generic (PLEG): container finished" podID="6abe5746-4718-4d28-88dc-5b9a54999869" containerID="9a645b2b558259c44ec3f2d04c256e92656e9eaa042da54d1ebdd05b56dfb02a" exitCode=0
Oct 30 19:19:55.591792 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.591608    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8" event={"ID":"6abe5746-4718-4d28-88dc-5b9a54999869","Type":"ContainerDied","Data":"9a645b2b558259c44ec3f2d04c256e92656e9eaa042da54d1ebdd05b56dfb02a"}
Oct 30 19:19:55.592012 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.591993    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8" event={"ID":"6abe5746-4718-4d28-88dc-5b9a54999869","Type":"ContainerDied","Data":"252635cefba26fbb94989bbd832917d16c1acf26d37afc868f78f9afd76ec619"}
Oct 30 19:19:55.592214 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.592154    4372 scope.go:122] "RemoveContainer" containerID="9a645b2b558259c44ec3f2d04c256e92656e9eaa042da54d1ebdd05b56dfb02a"
Oct 30 19:19:55.592454 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.591702    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"
Oct 30 19:19:55.596679 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.596655    4372 generic.go:356] "Generic (PLEG): container finished" podID="e09ce3e3-0b45-4f34-926f-a6a067ef44a4" containerID="d58740b9ee150659db9d8b2d41285d66f9a3955705bbc7d81224a29a7f48f4aa" exitCode=0
Oct 30 19:19:55.596889 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.596871    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6" event={"ID":"e09ce3e3-0b45-4f34-926f-a6a067ef44a4","Type":"ContainerDied","Data":"d58740b9ee150659db9d8b2d41285d66f9a3955705bbc7d81224a29a7f48f4aa"}
Oct 30 19:19:55.612741 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.612705    4372 scope.go:122] "RemoveContainer" containerID="9a645b2b558259c44ec3f2d04c256e92656e9eaa042da54d1ebdd05b56dfb02a"
Oct 30 19:19:55.613182 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:19:55.613107    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a645b2b558259c44ec3f2d04c256e92656e9eaa042da54d1ebdd05b56dfb02a\": container with ID starting with 9a645b2b558259c44ec3f2d04c256e92656e9eaa042da54d1ebdd05b56dfb02a not found: ID does not exist" containerID="9a645b2b558259c44ec3f2d04c256e92656e9eaa042da54d1ebdd05b56dfb02a"
Oct 30 19:19:55.613285 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.613197    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a645b2b558259c44ec3f2d04c256e92656e9eaa042da54d1ebdd05b56dfb02a"} err="failed to get container status \"9a645b2b558259c44ec3f2d04c256e92656e9eaa042da54d1ebdd05b56dfb02a\": rpc error: code = NotFound desc = could not find container \"9a645b2b558259c44ec3f2d04c256e92656e9eaa042da54d1ebdd05b56dfb02a\": container with ID starting with 9a645b2b558259c44ec3f2d04c256e92656e9eaa042da54d1ebdd05b56dfb02a not found: ID does not exist"
Oct 30 19:19:55.628456 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.627431    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"]
Oct 30 19:19:55.632563 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.632504    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-de4bda/payload-processing-external-model-e2e-extmod-a-de4bda-6b59wq7d8"]
Oct 30 19:19:55.764494 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.764457    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:55.858991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.858961    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-routing-overlay-volume\") pod \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") "
Oct 30 19:19:55.859190 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.859115    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-plugins-config-volume\") pod \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") "
Oct 30 19:19:55.859370 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.859350    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-kube-api-access-vr8hw\" (UniqueName: \"kubernetes.io/projected/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-kube-api-access-vr8hw\") pod \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") "
Oct 30 19:19:55.859415 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.859400    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-provider-credentials\") pod \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\" (UID: \"e09ce3e3-0b45-4f34-926f-a6a067ef44a4\") "
Oct 30 19:19:55.859504 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.859399    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-routing-overlay-volume" pod "e09ce3e3-0b45-4f34-926f-a6a067ef44a4" (UID: "e09ce3e3-0b45-4f34-926f-a6a067ef44a4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:55.859560 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.859473    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-plugins-config-volume" pod "e09ce3e3-0b45-4f34-926f-a6a067ef44a4" (UID: "e09ce3e3-0b45-4f34-926f-a6a067ef44a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:55.860084 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.860018    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:55.860084 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.860041    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:55.861659 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.861639    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-provider-credentials" pod "e09ce3e3-0b45-4f34-926f-a6a067ef44a4" (UID: "e09ce3e3-0b45-4f34-926f-a6a067ef44a4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:55.861794 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.861668    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-kube-api-access-vr8hw" pod "e09ce3e3-0b45-4f34-926f-a6a067ef44a4" (UID: "e09ce3e3-0b45-4f34-926f-a6a067ef44a4"). InnerVolumeSpecName "kube-api-access-vr8hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:55.961484 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.961429    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr8hw\" (UniqueName: \"kubernetes.io/projected/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-kube-api-access-vr8hw\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:55.961484 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:55.961454    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e09ce3e3-0b45-4f34-926f-a6a067ef44a4-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:56.267009 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:56.266927    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6abe5746-4718-4d28-88dc-5b9a54999869" path="/var/lib/kubelet/pods/6abe5746-4718-4d28-88dc-5b9a54999869/volumes"
Oct 30 19:19:56.304719 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:56.304690    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:19:56.304829 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:56.304753    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:19:56.606291 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:56.606254    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"
Oct 30 19:19:56.606496 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:56.606282    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6" event={"ID":"e09ce3e3-0b45-4f34-926f-a6a067ef44a4","Type":"ContainerDied","Data":"9b9fa2531a070cefefde9c9dd9e726b862aadd33cf47b83755e5b346f74b093a"}
Oct 30 19:19:56.606496 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:56.606356    4372 scope.go:122] "RemoveContainer" containerID="d58740b9ee150659db9d8b2d41285d66f9a3955705bbc7d81224a29a7f48f4aa"
Oct 30 19:19:56.635645 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:56.635614    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"]
Oct 30 19:19:56.639491 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:56.639468    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c6027/payload-processing-external-model-e2e-extmod-b-7c6027-5bdbhc9x6"]
Oct 30 19:19:57.697181 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:19:57.697143    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:19:58.267239 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:19:58.267211    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e09ce3e3-0b45-4f34-926f-a6a067ef44a4" path="/var/lib/kubelet/pods/e09ce3e3-0b45-4f34-926f-a6a067ef44a4/volumes"
Oct 30 19:20:00.573661 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.573627    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:20:00.642213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.642061    4372 generic.go:356] "Generic (PLEG): container finished" podID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerID="34914257f0fe610ae9da5ecb927e9b5e4e57f23f1acde2a2d9b94e875c1345a1" exitCode=0
Oct 30 19:20:00.642213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.642142    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"
Oct 30 19:20:00.642213 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.642163    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" event={"ID":"a6b96f6e-b871-499b-ab8f-18fffaa3ece2","Type":"ContainerDied","Data":"34914257f0fe610ae9da5ecb927e9b5e4e57f23f1acde2a2d9b94e875c1345a1"}
Oct 30 19:20:00.642513 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.642236    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd" event={"ID":"a6b96f6e-b871-499b-ab8f-18fffaa3ece2","Type":"ContainerDied","Data":"cf39e7a110ac0c7d391b2db897f3fc5279c6b9fa21f32ca669f37c7f7abc2720"}
Oct 30 19:20:00.642513 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.642270    4372 scope.go:122] "RemoveContainer" containerID="34914257f0fe610ae9da5ecb927e9b5e4e57f23f1acde2a2d9b94e875c1345a1"
Oct 30 19:20:00.656542 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.656518    4372 scope.go:122] "RemoveContainer" containerID="c1d6b783d5f6c51bb463972000e9bc0b29fa67bd71004c563bde6a647dcbe247"
Oct 30 19:20:00.657186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.657157    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-credential-socket\") pod \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") "
Oct 30 19:20:00.657314 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.657240    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istiod-ca-cert\") pod \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") "
Oct 30 19:20:00.657391 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.657318    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-token\" (UniqueName: \"kubernetes.io/projected/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-token\") pod \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") "
Oct 30 19:20:00.657461 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.657393    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-podinfo\") pod \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") "
Oct 30 19:20:00.657526 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.657500    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-envoy\") pod \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") "
Oct 30 19:20:00.657591 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.657554    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istiod-ca-cert" pod "a6b96f6e-b871-499b-ab8f-18fffaa3ece2" (UID: "a6b96f6e-b871-499b-ab8f-18fffaa3ece2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:00.657591 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.657563    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-data\") pod \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") "
Oct 30 19:20:00.657705 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.657671    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-workload-socket\") pod \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") "
Oct 30 19:20:00.657774 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.657722    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-kube-api-access-85hht\" (UniqueName: \"kubernetes.io/projected/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-kube-api-access-85hht\") pod \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") "
Oct 30 19:20:00.657844 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.657827    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-workload-certs\") pod \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\" (UID: \"a6b96f6e-b871-499b-ab8f-18fffaa3ece2\") "
Oct 30 19:20:00.658202 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.658174    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-credential-socket" pod "a6b96f6e-b871-499b-ab8f-18fffaa3ece2" (UID: "a6b96f6e-b871-499b-ab8f-18fffaa3ece2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:00.658468 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.658443    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-workload-certs" pod "a6b96f6e-b871-499b-ab8f-18fffaa3ece2" (UID: "a6b96f6e-b871-499b-ab8f-18fffaa3ece2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:00.658705 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.658677    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:00.658845 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.658710    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:00.658931 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.658897    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-workload-socket" pod "a6b96f6e-b871-499b-ab8f-18fffaa3ece2" (UID: "a6b96f6e-b871-499b-ab8f-18fffaa3ece2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:00.658996 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.658923    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-data" pod "a6b96f6e-b871-499b-ab8f-18fffaa3ece2" (UID: "a6b96f6e-b871-499b-ab8f-18fffaa3ece2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:00.661261 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.661230    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-envoy" pod "a6b96f6e-b871-499b-ab8f-18fffaa3ece2" (UID: "a6b96f6e-b871-499b-ab8f-18fffaa3ece2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:00.661908 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.661884    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-podinfo" pod "a6b96f6e-b871-499b-ab8f-18fffaa3ece2" (UID: "a6b96f6e-b871-499b-ab8f-18fffaa3ece2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:20:00.662001 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.661913    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-kube-api-access-85hht" pod "a6b96f6e-b871-499b-ab8f-18fffaa3ece2" (UID: "a6b96f6e-b871-499b-ab8f-18fffaa3ece2"). InnerVolumeSpecName "kube-api-access-85hht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:00.663528 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.663509    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-token" pod "a6b96f6e-b871-499b-ab8f-18fffaa3ece2" (UID: "a6b96f6e-b871-499b-ab8f-18fffaa3ece2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:00.693660 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.693619    4372 scope.go:122] "RemoveContainer" containerID="34914257f0fe610ae9da5ecb927e9b5e4e57f23f1acde2a2d9b94e875c1345a1"
Oct 30 19:20:00.693927 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:20:00.693907    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34914257f0fe610ae9da5ecb927e9b5e4e57f23f1acde2a2d9b94e875c1345a1\": container with ID starting with 34914257f0fe610ae9da5ecb927e9b5e4e57f23f1acde2a2d9b94e875c1345a1 not found: ID does not exist" containerID="34914257f0fe610ae9da5ecb927e9b5e4e57f23f1acde2a2d9b94e875c1345a1"
Oct 30 19:20:00.693972 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.693935    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34914257f0fe610ae9da5ecb927e9b5e4e57f23f1acde2a2d9b94e875c1345a1"} err="failed to get container status \"34914257f0fe610ae9da5ecb927e9b5e4e57f23f1acde2a2d9b94e875c1345a1\": rpc error: code = NotFound desc = could not find container \"34914257f0fe610ae9da5ecb927e9b5e4e57f23f1acde2a2d9b94e875c1345a1\": container with ID starting with 34914257f0fe610ae9da5ecb927e9b5e4e57f23f1acde2a2d9b94e875c1345a1 not found: ID does not exist"
Oct 30 19:20:00.693972 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.693950    4372 scope.go:122] "RemoveContainer" containerID="c1d6b783d5f6c51bb463972000e9bc0b29fa67bd71004c563bde6a647dcbe247"
Oct 30 19:20:00.694245 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:20:00.694225    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d6b783d5f6c51bb463972000e9bc0b29fa67bd71004c563bde6a647dcbe247\": container with ID starting with c1d6b783d5f6c51bb463972000e9bc0b29fa67bd71004c563bde6a647dcbe247 not found: ID does not exist" containerID="c1d6b783d5f6c51bb463972000e9bc0b29fa67bd71004c563bde6a647dcbe247"
Oct 30 19:20:00.694340 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.694249    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d6b783d5f6c51bb463972000e9bc0b29fa67bd71004c563bde6a647dcbe247"} err="failed to get container status \"c1d6b783d5f6c51bb463972000e9bc0b29fa67bd71004c563bde6a647dcbe247\": rpc error: code = NotFound desc = could not find container \"c1d6b783d5f6c51bb463972000e9bc0b29fa67bd71004c563bde6a647dcbe247\": container with ID starting with c1d6b783d5f6c51bb463972000e9bc0b29fa67bd71004c563bde6a647dcbe247 not found: ID does not exist"
Oct 30 19:20:00.760067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.760039    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:00.760067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.760064    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:00.760217 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.760073    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:00.760217 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.760082    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:00.760217 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.760091    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85hht\" (UniqueName: \"kubernetes.io/projected/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-kube-api-access-85hht\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:00.760217 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.760099    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:00.760217 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:00.760108    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b96f6e-b871-499b-ab8f-18fffaa3ece2-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:01.011640 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:01.011605    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"]
Oct 30 19:20:01.021453 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:01.021416    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e8c554-openshift-default-5dc548fd69-xmcfd"]
Oct 30 19:20:02.025707 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:02.025664    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"]
Oct 30 19:20:02.026237 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:02.025933    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl" podUID="2e3a7f70-606a-40b1-b541-e279a8438a32" containerName="istio-proxy" containerID="cri-o://b6bbfee57bbc93240a048de9058cee48dad0b941a84c815ca05efae66bc6457d" gracePeriod=30
Oct 30 19:20:02.312205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:02.311429    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" path="/var/lib/kubelet/pods/a6b96f6e-b871-499b-ab8f-18fffaa3ece2/volumes"
Oct 30 19:20:02.976702 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:02.976663    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:20:02.976878 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:02.976730    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl" podUID="2e3a7f70-606a-40b1-b541-e279a8438a32" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:20:05.840502 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:05.840470    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"]
Oct 30 19:20:05.874663 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:05.874620    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"]
Oct 30 19:20:07.314990 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.314932    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:20:07.405181 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.405147    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3a7f70-606a-40b1-b541-e279a8438a32-kube-api-access-zl9hr\" (UniqueName: \"kubernetes.io/projected/2e3a7f70-606a-40b1-b541-e279a8438a32-kube-api-access-zl9hr\") pod \"2e3a7f70-606a-40b1-b541-e279a8438a32\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") "
Oct 30 19:20:07.405350 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.405214    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-podinfo\") pod \"2e3a7f70-606a-40b1-b541-e279a8438a32\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") "
Oct 30 19:20:07.405350 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.405246    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-credential-socket\") pod \"2e3a7f70-606a-40b1-b541-e279a8438a32\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") "
Oct 30 19:20:07.405350 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.405275    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3a7f70-606a-40b1-b541-e279a8438a32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3a7f70-606a-40b1-b541-e279a8438a32-istiod-ca-cert\") pod \"2e3a7f70-606a-40b1-b541-e279a8438a32\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") "
Oct 30 19:20:07.405525 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.405393    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-token\" (UniqueName: \"kubernetes.io/projected/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-token\") pod \"2e3a7f70-606a-40b1-b541-e279a8438a32\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") "
Oct 30 19:20:07.405525 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.405442    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-data\") pod \"2e3a7f70-606a-40b1-b541-e279a8438a32\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") "
Oct 30 19:20:07.405525 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.405467    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-workload-certs\") pod \"2e3a7f70-606a-40b1-b541-e279a8438a32\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") "
Oct 30 19:20:07.405525 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.405521    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-envoy\") pod \"2e3a7f70-606a-40b1-b541-e279a8438a32\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") "
Oct 30 19:20:07.405751 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.405570    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-workload-socket\") pod \"2e3a7f70-606a-40b1-b541-e279a8438a32\" (UID: \"2e3a7f70-606a-40b1-b541-e279a8438a32\") "
Oct 30 19:20:07.406206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.405562    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-credential-socket" pod "2e3a7f70-606a-40b1-b541-e279a8438a32" (UID: "2e3a7f70-606a-40b1-b541-e279a8438a32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:07.406206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.405786    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-workload-certs" pod "2e3a7f70-606a-40b1-b541-e279a8438a32" (UID: "2e3a7f70-606a-40b1-b541-e279a8438a32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:07.406206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.405820    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3a7f70-606a-40b1-b541-e279a8438a32-istiod-ca-cert" pod "2e3a7f70-606a-40b1-b541-e279a8438a32" (UID: "2e3a7f70-606a-40b1-b541-e279a8438a32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:07.406480 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.406259    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-data" pod "2e3a7f70-606a-40b1-b541-e279a8438a32" (UID: "2e3a7f70-606a-40b1-b541-e279a8438a32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:07.406925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.406760    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-workload-socket" pod "2e3a7f70-606a-40b1-b541-e279a8438a32" (UID: "2e3a7f70-606a-40b1-b541-e279a8438a32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:07.406925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.406786    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.406925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.406810    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3a7f70-606a-40b1-b541-e279a8438a32-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.406925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.406825    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.406925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.406841    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.408271 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.408246    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-podinfo" pod "2e3a7f70-606a-40b1-b541-e279a8438a32" (UID: "2e3a7f70-606a-40b1-b541-e279a8438a32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:20:07.408391 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.408339    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3a7f70-606a-40b1-b541-e279a8438a32-kube-api-access-zl9hr" pod "2e3a7f70-606a-40b1-b541-e279a8438a32" (UID: "2e3a7f70-606a-40b1-b541-e279a8438a32"). InnerVolumeSpecName "kube-api-access-zl9hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:07.408879 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.408857    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-envoy" pod "2e3a7f70-606a-40b1-b541-e279a8438a32" (UID: "2e3a7f70-606a-40b1-b541-e279a8438a32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:07.409226 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.409203    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-token" pod "2e3a7f70-606a-40b1-b541-e279a8438a32" (UID: "2e3a7f70-606a-40b1-b541-e279a8438a32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:07.507685 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.507660    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl9hr\" (UniqueName: \"kubernetes.io/projected/2e3a7f70-606a-40b1-b541-e279a8438a32-kube-api-access-zl9hr\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.507685 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.507685    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.507814 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.507695    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.507814 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.507704    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.507814 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.507712    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3a7f70-606a-40b1-b541-e279a8438a32-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:07.700580 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.700548    4372 generic.go:356] "Generic (PLEG): container finished" podID="2e3a7f70-606a-40b1-b541-e279a8438a32" containerID="b6bbfee57bbc93240a048de9058cee48dad0b941a84c815ca05efae66bc6457d" exitCode=0
Oct 30 19:20:07.700701 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.700595    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"
Oct 30 19:20:07.700701 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.700635    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl" event={"ID":"2e3a7f70-606a-40b1-b541-e279a8438a32","Type":"ContainerDied","Data":"b6bbfee57bbc93240a048de9058cee48dad0b941a84c815ca05efae66bc6457d"}
Oct 30 19:20:07.700701 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.700694    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl" event={"ID":"2e3a7f70-606a-40b1-b541-e279a8438a32","Type":"ContainerDied","Data":"b6e6826e9b160ed219d377ffed0be8dcfce6a085ebdf059a19f4047f0bb3eaf0"}
Oct 30 19:20:07.700823 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.700719    4372 scope.go:122] "RemoveContainer" containerID="b6bbfee57bbc93240a048de9058cee48dad0b941a84c815ca05efae66bc6457d"
Oct 30 19:20:07.712426 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.712407    4372 scope.go:122] "RemoveContainer" containerID="b6bbfee57bbc93240a048de9058cee48dad0b941a84c815ca05efae66bc6457d"
Oct 30 19:20:07.712698 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:20:07.712678    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6bbfee57bbc93240a048de9058cee48dad0b941a84c815ca05efae66bc6457d\": container with ID starting with b6bbfee57bbc93240a048de9058cee48dad0b941a84c815ca05efae66bc6457d not found: ID does not exist" containerID="b6bbfee57bbc93240a048de9058cee48dad0b941a84c815ca05efae66bc6457d"
Oct 30 19:20:07.712783 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.712708    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6bbfee57bbc93240a048de9058cee48dad0b941a84c815ca05efae66bc6457d"} err="failed to get container status \"b6bbfee57bbc93240a048de9058cee48dad0b941a84c815ca05efae66bc6457d\": rpc error: code = NotFound desc = could not find container \"b6bbfee57bbc93240a048de9058cee48dad0b941a84c815ca05efae66bc6457d\": container with ID starting with b6bbfee57bbc93240a048de9058cee48dad0b941a84c815ca05efae66bc6457d not found: ID does not exist"
Oct 30 19:20:07.731868 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.731845    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"]
Oct 30 19:20:07.738340 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:07.738320    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e77b1d-openshift-default-69dfd9fbb5-zt8zl"]
Oct 30 19:20:08.286151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:08.285440    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3a7f70-606a-40b1-b541-e279a8438a32" path="/var/lib/kubelet/pods/2e3a7f70-606a-40b1-b541-e279a8438a32/volumes"
Oct 30 19:20:10.840967 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:10.840923    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4" podUID="f681d0ac-6413-4ccb-88ba-3e0e278bd663" containerName="payload-processing" containerID="cri-o://2c4cd7e63cb5481b2a01cc05ae4d1aa55c721d91ac91b2487e9152921e5dd469" gracePeriod=25
Oct 30 19:20:10.875315 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:10.875265    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6" podUID="4f57597b-6f1c-47e2-9891-5432b04e8fb1" containerName="payload-pre-processing" containerID="cri-o://ef9d8561830b85bb2aa80545eb768243264c44200dca730cf3e72ec52c27edda" gracePeriod=25
Oct 30 19:20:13.572950 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:13.572816    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4" podUID="f681d0ac-6413-4ccb-88ba-3e0e278bd663" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:13.576101 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:13.574627    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6" podUID="4f57597b-6f1c-47e2-9891-5432b04e8fb1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:23.563465 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:23.563381    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6" podUID="4f57597b-6f1c-47e2-9891-5432b04e8fb1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:23.563974 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:23.563381    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4" podUID="f681d0ac-6413-4ccb-88ba-3e0e278bd663" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:25.637992 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:25.637951    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"]
Oct 30 19:20:25.638525 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:25.638268    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerName="istio-proxy" containerID="cri-o://69fcbf596531795b3d5bce24f7d5093c9c8b2287f646c3236013fdbd8288a9ba" gracePeriod=30
Oct 30 19:20:27.698195 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:20:27.698159    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:20:27.763193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:27.763153    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:20:27.764271 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:27.764211    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:20:29.543179 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:29.543142    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:20:30.942500 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:30.942472    4372 generic.go:356] "Generic (PLEG): container finished" podID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerID="69fcbf596531795b3d5bce24f7d5093c9c8b2287f646c3236013fdbd8288a9ba" exitCode=0
Oct 30 19:20:30.942859 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:30.942530    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" event={"ID":"9313e8c4-ec9e-4735-b58a-ce18fa0a692b","Type":"ContainerDied","Data":"69fcbf596531795b3d5bce24f7d5093c9c8b2287f646c3236013fdbd8288a9ba"}
Oct 30 19:20:30.942859 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:30.942576    4372 scope.go:122] "RemoveContainer" containerID="04a5f256ae5398fcb41558cdad9e438580b3aaacdd46eb94dd7ec1356d055057"
Oct 30 19:20:31.019148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.018345    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:20:31.119030 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.119001    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-kube-api-access-s44wk\" (UniqueName: \"kubernetes.io/projected/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-kube-api-access-s44wk\") pod \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") "
Oct 30 19:20:31.119298 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.119092    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-workload-certs\") pod \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") "
Oct 30 19:20:31.119298 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.119152    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-token\" (UniqueName: \"kubernetes.io/projected/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-token\") pod \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") "
Oct 30 19:20:31.119509 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.119387    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-credential-socket\") pod \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") "
Oct 30 19:20:31.119509 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.119420    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istiod-ca-cert\") pod \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") "
Oct 30 19:20:31.119671 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.119537    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-podinfo\") pod \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") "
Oct 30 19:20:31.119671 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.119628    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-data\") pod \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") "
Oct 30 19:20:31.119826 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.119785    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-envoy\") pod \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") "
Oct 30 19:20:31.119906 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.119878    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-workload-socket\") pod \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\" (UID: \"9313e8c4-ec9e-4735-b58a-ce18fa0a692b\") "
Oct 30 19:20:31.120670 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.120641    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-workload-certs" pod "9313e8c4-ec9e-4735-b58a-ce18fa0a692b" (UID: "9313e8c4-ec9e-4735-b58a-ce18fa0a692b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:31.121045 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.121016    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-data" pod "9313e8c4-ec9e-4735-b58a-ce18fa0a692b" (UID: "9313e8c4-ec9e-4735-b58a-ce18fa0a692b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:31.121186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.121090    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-workload-socket" pod "9313e8c4-ec9e-4735-b58a-ce18fa0a692b" (UID: "9313e8c4-ec9e-4735-b58a-ce18fa0a692b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:31.121431 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.121402    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-credential-socket" pod "9313e8c4-ec9e-4735-b58a-ce18fa0a692b" (UID: "9313e8c4-ec9e-4735-b58a-ce18fa0a692b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:31.121582 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.121424    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istiod-ca-cert" pod "9313e8c4-ec9e-4735-b58a-ce18fa0a692b" (UID: "9313e8c4-ec9e-4735-b58a-ce18fa0a692b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:31.123864 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.123841    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-kube-api-access-s44wk" pod "9313e8c4-ec9e-4735-b58a-ce18fa0a692b" (UID: "9313e8c4-ec9e-4735-b58a-ce18fa0a692b"). InnerVolumeSpecName "kube-api-access-s44wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:31.124003 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.123976    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-podinfo" pod "9313e8c4-ec9e-4735-b58a-ce18fa0a692b" (UID: "9313e8c4-ec9e-4735-b58a-ce18fa0a692b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:20:31.126065 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.126030    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-envoy" pod "9313e8c4-ec9e-4735-b58a-ce18fa0a692b" (UID: "9313e8c4-ec9e-4735-b58a-ce18fa0a692b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:31.127365 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.127340    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-token" pod "9313e8c4-ec9e-4735-b58a-ce18fa0a692b" (UID: "9313e8c4-ec9e-4735-b58a-ce18fa0a692b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:31.192308 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.192273    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"
Oct 30 19:20:31.217363 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.217328    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"
Oct 30 19:20:31.221892 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.221865    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.221892 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.221890    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.222085 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.221901    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.222085 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.221911    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.222085 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.221921    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s44wk\" (UniqueName: \"kubernetes.io/projected/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-kube-api-access-s44wk\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.222085 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.221931    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.222085 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.221939    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.222085 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.221948    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.222085 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.221957    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9313e8c4-ec9e-4735-b58a-ce18fa0a692b-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.322562 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.322529    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f57597b-6f1c-47e2-9891-5432b04e8fb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f57597b-6f1c-47e2-9891-5432b04e8fb1-plugins-config-volume\") pod \"4f57597b-6f1c-47e2-9891-5432b04e8fb1\" (UID: \"4f57597b-6f1c-47e2-9891-5432b04e8fb1\") "
Oct 30 19:20:31.322750 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.322722    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f681d0ac-6413-4ccb-88ba-3e0e278bd663-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f681d0ac-6413-4ccb-88ba-3e0e278bd663-plugins-config-volume\") pod \"f681d0ac-6413-4ccb-88ba-3e0e278bd663\" (UID: \"f681d0ac-6413-4ccb-88ba-3e0e278bd663\") "
Oct 30 19:20:31.322874 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.322859    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f681d0ac-6413-4ccb-88ba-3e0e278bd663-kube-api-access-96p5j\" (UniqueName: \"kubernetes.io/projected/f681d0ac-6413-4ccb-88ba-3e0e278bd663-kube-api-access-96p5j\") pod \"f681d0ac-6413-4ccb-88ba-3e0e278bd663\" (UID: \"f681d0ac-6413-4ccb-88ba-3e0e278bd663\") "
Oct 30 19:20:31.322958 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.322895    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f57597b-6f1c-47e2-9891-5432b04e8fb1-kube-api-access-4mdfc\" (UniqueName: \"kubernetes.io/projected/4f57597b-6f1c-47e2-9891-5432b04e8fb1-kube-api-access-4mdfc\") pod \"4f57597b-6f1c-47e2-9891-5432b04e8fb1\" (UID: \"4f57597b-6f1c-47e2-9891-5432b04e8fb1\") "
Oct 30 19:20:31.323023 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.322949    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f57597b-6f1c-47e2-9891-5432b04e8fb1-plugins-config-volume" pod "4f57597b-6f1c-47e2-9891-5432b04e8fb1" (UID: "4f57597b-6f1c-47e2-9891-5432b04e8fb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:31.323100 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.323082    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f681d0ac-6413-4ccb-88ba-3e0e278bd663-plugins-config-volume" pod "f681d0ac-6413-4ccb-88ba-3e0e278bd663" (UID: "f681d0ac-6413-4ccb-88ba-3e0e278bd663"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:31.323418 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.323396    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f57597b-6f1c-47e2-9891-5432b04e8fb1-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.323486 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.323422    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f681d0ac-6413-4ccb-88ba-3e0e278bd663-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.325297 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.325277    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f681d0ac-6413-4ccb-88ba-3e0e278bd663-kube-api-access-96p5j" pod "f681d0ac-6413-4ccb-88ba-3e0e278bd663" (UID: "f681d0ac-6413-4ccb-88ba-3e0e278bd663"). InnerVolumeSpecName "kube-api-access-96p5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:31.325380 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.325306    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f57597b-6f1c-47e2-9891-5432b04e8fb1-kube-api-access-4mdfc" pod "4f57597b-6f1c-47e2-9891-5432b04e8fb1" (UID: "4f57597b-6f1c-47e2-9891-5432b04e8fb1"). InnerVolumeSpecName "kube-api-access-4mdfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:31.425294 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.425259    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96p5j\" (UniqueName: \"kubernetes.io/projected/f681d0ac-6413-4ccb-88ba-3e0e278bd663-kube-api-access-96p5j\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.425294 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.425295    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mdfc\" (UniqueName: \"kubernetes.io/projected/4f57597b-6f1c-47e2-9891-5432b04e8fb1-kube-api-access-4mdfc\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.950960 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.950869    4372 generic.go:356] "Generic (PLEG): container finished" podID="f681d0ac-6413-4ccb-88ba-3e0e278bd663" containerID="2c4cd7e63cb5481b2a01cc05ae4d1aa55c721d91ac91b2487e9152921e5dd469" exitCode=0
Oct 30 19:20:31.950960 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.950921    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"
Oct 30 19:20:31.951525 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.950955    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4" event={"ID":"f681d0ac-6413-4ccb-88ba-3e0e278bd663","Type":"ContainerDied","Data":"2c4cd7e63cb5481b2a01cc05ae4d1aa55c721d91ac91b2487e9152921e5dd469"}
Oct 30 19:20:31.951525 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.951022    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4" event={"ID":"f681d0ac-6413-4ccb-88ba-3e0e278bd663","Type":"ContainerDied","Data":"f07c0660e10deb02b7a42e7439d3e2be8107639e43692cd6593a3a118054ec97"}
Oct 30 19:20:31.951525 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.951056    4372 scope.go:122] "RemoveContainer" containerID="2c4cd7e63cb5481b2a01cc05ae4d1aa55c721d91ac91b2487e9152921e5dd469"
Oct 30 19:20:31.952600 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.952582    4372 generic.go:356] "Generic (PLEG): container finished" podID="4f57597b-6f1c-47e2-9891-5432b04e8fb1" containerID="ef9d8561830b85bb2aa80545eb768243264c44200dca730cf3e72ec52c27edda" exitCode=0
Oct 30 19:20:31.952669 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.952637    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"
Oct 30 19:20:31.952718 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.952671    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6" event={"ID":"4f57597b-6f1c-47e2-9891-5432b04e8fb1","Type":"ContainerDied","Data":"ef9d8561830b85bb2aa80545eb768243264c44200dca730cf3e72ec52c27edda"}
Oct 30 19:20:31.952767 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.952719    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6" event={"ID":"4f57597b-6f1c-47e2-9891-5432b04e8fb1","Type":"ContainerDied","Data":"6276cf05c3bc622a26a8318156ac7142a0a325169597bd48d74fcf30854ed1dd"}
Oct 30 19:20:31.954264 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.954243    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"
Oct 30 19:20:31.954371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.954251    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj" event={"ID":"9313e8c4-ec9e-4735-b58a-ce18fa0a692b","Type":"ContainerDied","Data":"1de6da698bc66de6d6b0b63ca4382be5dbcbfc8fd8ef8624cb656ce5f6a64d83"}
Oct 30 19:20:31.964759 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.964742    4372 scope.go:122] "RemoveContainer" containerID="2c4cd7e63cb5481b2a01cc05ae4d1aa55c721d91ac91b2487e9152921e5dd469"
Oct 30 19:20:31.964988 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:20:31.964969    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c4cd7e63cb5481b2a01cc05ae4d1aa55c721d91ac91b2487e9152921e5dd469\": container with ID starting with 2c4cd7e63cb5481b2a01cc05ae4d1aa55c721d91ac91b2487e9152921e5dd469 not found: ID does not exist" containerID="2c4cd7e63cb5481b2a01cc05ae4d1aa55c721d91ac91b2487e9152921e5dd469"
Oct 30 19:20:31.965041 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.964996    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c4cd7e63cb5481b2a01cc05ae4d1aa55c721d91ac91b2487e9152921e5dd469"} err="failed to get container status \"2c4cd7e63cb5481b2a01cc05ae4d1aa55c721d91ac91b2487e9152921e5dd469\": rpc error: code = NotFound desc = could not find container \"2c4cd7e63cb5481b2a01cc05ae4d1aa55c721d91ac91b2487e9152921e5dd469\": container with ID starting with 2c4cd7e63cb5481b2a01cc05ae4d1aa55c721d91ac91b2487e9152921e5dd469 not found: ID does not exist"
Oct 30 19:20:31.965041 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.965014    4372 scope.go:122] "RemoveContainer" containerID="ef9d8561830b85bb2aa80545eb768243264c44200dca730cf3e72ec52c27edda"
Oct 30 19:20:31.975020 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.975001    4372 scope.go:122] "RemoveContainer" containerID="ef9d8561830b85bb2aa80545eb768243264c44200dca730cf3e72ec52c27edda"
Oct 30 19:20:31.975347 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:20:31.975321    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9d8561830b85bb2aa80545eb768243264c44200dca730cf3e72ec52c27edda\": container with ID starting with ef9d8561830b85bb2aa80545eb768243264c44200dca730cf3e72ec52c27edda not found: ID does not exist" containerID="ef9d8561830b85bb2aa80545eb768243264c44200dca730cf3e72ec52c27edda"
Oct 30 19:20:31.975431 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.975352    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9d8561830b85bb2aa80545eb768243264c44200dca730cf3e72ec52c27edda"} err="failed to get container status \"ef9d8561830b85bb2aa80545eb768243264c44200dca730cf3e72ec52c27edda\": rpc error: code = NotFound desc = could not find container \"ef9d8561830b85bb2aa80545eb768243264c44200dca730cf3e72ec52c27edda\": container with ID starting with ef9d8561830b85bb2aa80545eb768243264c44200dca730cf3e72ec52c27edda not found: ID does not exist"
Oct 30 19:20:31.975431 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:31.975368    4372 scope.go:122] "RemoveContainer" containerID="69fcbf596531795b3d5bce24f7d5093c9c8b2287f646c3236013fdbd8288a9ba"
Oct 30 19:20:32.007671 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:32.007650    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"]
Oct 30 19:20:32.012436 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:32.012410    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4e94aa-fb88d8d7-5vcg6"]
Oct 30 19:20:32.027238 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:32.027216    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"]
Oct 30 19:20:32.033314 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:32.033292    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4e94aa-57485cf9fd-9dfq4"]
Oct 30 19:20:32.048527 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:32.048505    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"]
Oct 30 19:20:32.054439 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:32.054418    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-4e94aa-openshift-default-56c59f747c-vpclj"]
Oct 30 19:20:32.276383 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:32.276299    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f57597b-6f1c-47e2-9891-5432b04e8fb1" path="/var/lib/kubelet/pods/4f57597b-6f1c-47e2-9891-5432b04e8fb1/volumes"
Oct 30 19:20:32.276965 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:32.276946    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" path="/var/lib/kubelet/pods/9313e8c4-ec9e-4735-b58a-ce18fa0a692b/volumes"
Oct 30 19:20:32.277533 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:32.277513    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f681d0ac-6413-4ccb-88ba-3e0e278bd663" path="/var/lib/kubelet/pods/f681d0ac-6413-4ccb-88ba-3e0e278bd663/volumes"
Oct 30 19:20:33.141571 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:33.141527    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"]
Oct 30 19:20:33.142035 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:33.141788    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd" podUID="36220b42-54dc-4dd2-aac8-e44b93f61a55" containerName="istio-proxy" containerID="cri-o://bf4cf844c1d99bc99a86d755f0a3397cf9f583670311c9771aaba696c841e138" gracePeriod=30
Oct 30 19:20:33.807013 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:33.806975    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"]
Oct 30 19:20:33.807282 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:33.807255    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr" podUID="cf21565a-4381-4ed0-ad6c-015853c23d1e" containerName="istio-proxy" containerID="cri-o://b8233b629e22056cc8404c2bd383cdef89f87f035f630638b45638ab1495395d" gracePeriod=30
Oct 30 19:20:33.810844 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:33.810816    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"]
Oct 30 19:20:33.811066 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:33.811034    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh" podUID="e2284506-257f-4b40-a5c0-69282848c9b6" containerName="istio-proxy" containerID="cri-o://1fb60d5766ffae739154e091bbc4a5dee22f04ce5a1512d60187983196eede0c" gracePeriod=30
Oct 30 19:20:36.341647 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:36.341613    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"]
Oct 30 19:20:36.376318 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:36.376281    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"]
Oct 30 19:20:37.559058 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:37.559021    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:20:37.559487 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:37.559099    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr" podUID="cf21565a-4381-4ed0-ad6c-015853c23d1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:20:37.699322 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:37.699292    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:20:37.699484 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:37.699357    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh" podUID="e2284506-257f-4b40-a5c0-69282848c9b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:20:38.416658 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.416603    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 19:20:38.465408 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.465374    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-credential-socket\") pod \"36220b42-54dc-4dd2-aac8-e44b93f61a55\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") "
Oct 30 19:20:38.465552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.465471    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-podinfo\") pod \"36220b42-54dc-4dd2-aac8-e44b93f61a55\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") "
Oct 30 19:20:38.465552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.465513    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-workload-socket\") pod \"36220b42-54dc-4dd2-aac8-e44b93f61a55\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") "
Oct 30 19:20:38.465677 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.465577    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-token\" (UniqueName: \"kubernetes.io/projected/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-token\") pod \"36220b42-54dc-4dd2-aac8-e44b93f61a55\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") "
Oct 30 19:20:38.465677 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.465605    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36220b42-54dc-4dd2-aac8-e44b93f61a55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36220b42-54dc-4dd2-aac8-e44b93f61a55-istiod-ca-cert\") pod \"36220b42-54dc-4dd2-aac8-e44b93f61a55\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") "
Oct 30 19:20:38.465677 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.465631    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-credential-socket" pod "36220b42-54dc-4dd2-aac8-e44b93f61a55" (UID: "36220b42-54dc-4dd2-aac8-e44b93f61a55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:38.465844 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.465670    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-workload-certs\") pod \"36220b42-54dc-4dd2-aac8-e44b93f61a55\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") "
Oct 30 19:20:38.465844 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.465813    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36220b42-54dc-4dd2-aac8-e44b93f61a55-kube-api-access-b6wr9\" (UniqueName: \"kubernetes.io/projected/36220b42-54dc-4dd2-aac8-e44b93f61a55-kube-api-access-b6wr9\") pod \"36220b42-54dc-4dd2-aac8-e44b93f61a55\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") "
Oct 30 19:20:38.465977 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.465855    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-workload-socket" pod "36220b42-54dc-4dd2-aac8-e44b93f61a55" (UID: "36220b42-54dc-4dd2-aac8-e44b93f61a55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:38.465977 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.465928    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-envoy\") pod \"36220b42-54dc-4dd2-aac8-e44b93f61a55\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") "
Oct 30 19:20:38.466086 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.465979    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-data\") pod \"36220b42-54dc-4dd2-aac8-e44b93f61a55\" (UID: \"36220b42-54dc-4dd2-aac8-e44b93f61a55\") "
Oct 30 19:20:38.466086 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.466052    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-workload-certs" pod "36220b42-54dc-4dd2-aac8-e44b93f61a55" (UID: "36220b42-54dc-4dd2-aac8-e44b93f61a55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:38.466224 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.466189    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36220b42-54dc-4dd2-aac8-e44b93f61a55-istiod-ca-cert" pod "36220b42-54dc-4dd2-aac8-e44b93f61a55" (UID: "36220b42-54dc-4dd2-aac8-e44b93f61a55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:38.466777 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.466750    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-data" pod "36220b42-54dc-4dd2-aac8-e44b93f61a55" (UID: "36220b42-54dc-4dd2-aac8-e44b93f61a55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:38.466876 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.466834    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:38.466876 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.466858    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:38.466876 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.466872    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36220b42-54dc-4dd2-aac8-e44b93f61a55-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:38.467017 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.466888    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:38.469441 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.469408    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-envoy" pod "36220b42-54dc-4dd2-aac8-e44b93f61a55" (UID: "36220b42-54dc-4dd2-aac8-e44b93f61a55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:38.469541 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.469453    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36220b42-54dc-4dd2-aac8-e44b93f61a55-kube-api-access-b6wr9" pod "36220b42-54dc-4dd2-aac8-e44b93f61a55" (UID: "36220b42-54dc-4dd2-aac8-e44b93f61a55"). InnerVolumeSpecName "kube-api-access-b6wr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:38.469615 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.469580    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-token" pod "36220b42-54dc-4dd2-aac8-e44b93f61a55" (UID: "36220b42-54dc-4dd2-aac8-e44b93f61a55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:38.469681 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.469634    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-podinfo" pod "36220b42-54dc-4dd2-aac8-e44b93f61a55" (UID: "36220b42-54dc-4dd2-aac8-e44b93f61a55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:20:38.568014 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.567984    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:38.568014 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.568012    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:38.568403 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.568022    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:38.568403 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.568031    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36220b42-54dc-4dd2-aac8-e44b93f61a55-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:38.568403 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:38.568041    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6wr9\" (UniqueName: \"kubernetes.io/projected/36220b42-54dc-4dd2-aac8-e44b93f61a55-kube-api-access-b6wr9\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.062043 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.062006    4372 generic.go:356] "Generic (PLEG): container finished" podID="cf21565a-4381-4ed0-ad6c-015853c23d1e" containerID="b8233b629e22056cc8404c2bd383cdef89f87f035f630638b45638ab1495395d" exitCode=0
Oct 30 19:20:39.062259 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.062066    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr" event={"ID":"cf21565a-4381-4ed0-ad6c-015853c23d1e","Type":"ContainerDied","Data":"b8233b629e22056cc8404c2bd383cdef89f87f035f630638b45638ab1495395d"}
Oct 30 19:20:39.064558 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.064513    4372 generic.go:356] "Generic (PLEG): container finished" podID="36220b42-54dc-4dd2-aac8-e44b93f61a55" containerID="bf4cf844c1d99bc99a86d755f0a3397cf9f583670311c9771aaba696c841e138" exitCode=0
Oct 30 19:20:39.064704 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.064624    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"
Oct 30 19:20:39.064704 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.064623    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd" event={"ID":"36220b42-54dc-4dd2-aac8-e44b93f61a55","Type":"ContainerDied","Data":"bf4cf844c1d99bc99a86d755f0a3397cf9f583670311c9771aaba696c841e138"}
Oct 30 19:20:39.064861 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.064747    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd" event={"ID":"36220b42-54dc-4dd2-aac8-e44b93f61a55","Type":"ContainerDied","Data":"7aee46a91d6c1c138ceaddebbe76eb5461f53ea682560081d57dfe22717f0f0b"}
Oct 30 19:20:39.064861 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.064782    4372 scope.go:122] "RemoveContainer" containerID="bf4cf844c1d99bc99a86d755f0a3397cf9f583670311c9771aaba696c841e138"
Oct 30 19:20:39.066715 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.066590    4372 generic.go:356] "Generic (PLEG): container finished" podID="e2284506-257f-4b40-a5c0-69282848c9b6" containerID="1fb60d5766ffae739154e091bbc4a5dee22f04ce5a1512d60187983196eede0c" exitCode=0
Oct 30 19:20:39.066715 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.066667    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh" event={"ID":"e2284506-257f-4b40-a5c0-69282848c9b6","Type":"ContainerDied","Data":"1fb60d5766ffae739154e091bbc4a5dee22f04ce5a1512d60187983196eede0c"}
Oct 30 19:20:39.083223 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.083200    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:20:39.100360 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.090651    4372 scope.go:122] "RemoveContainer" containerID="dccce248504137e4bd9e5c375b4d844071c7f1692e23f9eb75e1c57e7b1b4ecb"
Oct 30 19:20:39.104212 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.104197    4372 scope.go:122] "RemoveContainer" containerID="bf4cf844c1d99bc99a86d755f0a3397cf9f583670311c9771aaba696c841e138"
Oct 30 19:20:39.104524 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:20:39.104496    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4cf844c1d99bc99a86d755f0a3397cf9f583670311c9771aaba696c841e138\": container with ID starting with bf4cf844c1d99bc99a86d755f0a3397cf9f583670311c9771aaba696c841e138 not found: ID does not exist" containerID="bf4cf844c1d99bc99a86d755f0a3397cf9f583670311c9771aaba696c841e138"
Oct 30 19:20:39.104613 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.104535    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4cf844c1d99bc99a86d755f0a3397cf9f583670311c9771aaba696c841e138"} err="failed to get container status \"bf4cf844c1d99bc99a86d755f0a3397cf9f583670311c9771aaba696c841e138\": rpc error: code = NotFound desc = could not find container \"bf4cf844c1d99bc99a86d755f0a3397cf9f583670311c9771aaba696c841e138\": container with ID starting with bf4cf844c1d99bc99a86d755f0a3397cf9f583670311c9771aaba696c841e138 not found: ID does not exist"
Oct 30 19:20:39.104613 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.104558    4372 scope.go:122] "RemoveContainer" containerID="dccce248504137e4bd9e5c375b4d844071c7f1692e23f9eb75e1c57e7b1b4ecb"
Oct 30 19:20:39.104854 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:20:39.104829    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dccce248504137e4bd9e5c375b4d844071c7f1692e23f9eb75e1c57e7b1b4ecb\": container with ID starting with dccce248504137e4bd9e5c375b4d844071c7f1692e23f9eb75e1c57e7b1b4ecb not found: ID does not exist" containerID="dccce248504137e4bd9e5c375b4d844071c7f1692e23f9eb75e1c57e7b1b4ecb"
Oct 30 19:20:39.104965 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.104856    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dccce248504137e4bd9e5c375b4d844071c7f1692e23f9eb75e1c57e7b1b4ecb"} err="failed to get container status \"dccce248504137e4bd9e5c375b4d844071c7f1692e23f9eb75e1c57e7b1b4ecb\": rpc error: code = NotFound desc = could not find container \"dccce248504137e4bd9e5c375b4d844071c7f1692e23f9eb75e1c57e7b1b4ecb\": container with ID starting with dccce248504137e4bd9e5c375b4d844071c7f1692e23f9eb75e1c57e7b1b4ecb not found: ID does not exist"
Oct 30 19:20:39.119090 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.119062    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"]
Oct 30 19:20:39.125197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.125174    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-gpgwd"]
Oct 30 19:20:39.146000 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.145977    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:20:39.177818 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.177794    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-envoy\") pod \"cf21565a-4381-4ed0-ad6c-015853c23d1e\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") "
Oct 30 19:20:39.177941 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.177830    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2284506-257f-4b40-a5c0-69282848c9b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2284506-257f-4b40-a5c0-69282848c9b6-istio-podinfo\") pod \"e2284506-257f-4b40-a5c0-69282848c9b6\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") "
Oct 30 19:20:39.177941 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.177865    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-credential-socket\") pod \"e2284506-257f-4b40-a5c0-69282848c9b6\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") "
Oct 30 19:20:39.177941 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.177892    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-workload-socket\") pod \"cf21565a-4381-4ed0-ad6c-015853c23d1e\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") "
Oct 30 19:20:39.178093 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.177979    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2284506-257f-4b40-a5c0-69282848c9b6-kube-api-access-bp5vc\" (UniqueName: \"kubernetes.io/projected/e2284506-257f-4b40-a5c0-69282848c9b6-kube-api-access-bp5vc\") pod \"e2284506-257f-4b40-a5c0-69282848c9b6\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") "
Oct 30 19:20:39.178093 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178011    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2284506-257f-4b40-a5c0-69282848c9b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2284506-257f-4b40-a5c0-69282848c9b6-istiod-ca-cert\") pod \"e2284506-257f-4b40-a5c0-69282848c9b6\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") "
Oct 30 19:20:39.178093 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178053    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-istio-envoy\") pod \"e2284506-257f-4b40-a5c0-69282848c9b6\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") "
Oct 30 19:20:39.178266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178116    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-credential-socket\") pod \"cf21565a-4381-4ed0-ad6c-015853c23d1e\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") "
Oct 30 19:20:39.178266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178190    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-workload-certs\") pod \"cf21565a-4381-4ed0-ad6c-015853c23d1e\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") "
Oct 30 19:20:39.178266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178225    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-workload-socket\") pod \"e2284506-257f-4b40-a5c0-69282848c9b6\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") "
Oct 30 19:20:39.178266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178243    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-credential-socket" pod "e2284506-257f-4b40-a5c0-69282848c9b6" (UID: "e2284506-257f-4b40-a5c0-69282848c9b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.178490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178295    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2284506-257f-4b40-a5c0-69282848c9b6-istio-token\" (UniqueName: \"kubernetes.io/projected/e2284506-257f-4b40-a5c0-69282848c9b6-istio-token\") pod \"e2284506-257f-4b40-a5c0-69282848c9b6\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") "
Oct 30 19:20:39.178490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178338    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-workload-socket" pod "cf21565a-4381-4ed0-ad6c-015853c23d1e" (UID: "cf21565a-4381-4ed0-ad6c-015853c23d1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.178490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178374    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-podinfo\") pod \"cf21565a-4381-4ed0-ad6c-015853c23d1e\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") "
Oct 30 19:20:39.178490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178410    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-workload-certs\") pod \"e2284506-257f-4b40-a5c0-69282848c9b6\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") "
Oct 30 19:20:39.178490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178477    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf21565a-4381-4ed0-ad6c-015853c23d1e-kube-api-access-j85jt\" (UniqueName: \"kubernetes.io/projected/cf21565a-4381-4ed0-ad6c-015853c23d1e-kube-api-access-j85jt\") pod \"cf21565a-4381-4ed0-ad6c-015853c23d1e\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") "
Oct 30 19:20:39.178793 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178498    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-credential-socket" pod "cf21565a-4381-4ed0-ad6c-015853c23d1e" (UID: "cf21565a-4381-4ed0-ad6c-015853c23d1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.178793 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178537    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-workload-socket" pod "e2284506-257f-4b40-a5c0-69282848c9b6" (UID: "e2284506-257f-4b40-a5c0-69282848c9b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.178793 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178545    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2284506-257f-4b40-a5c0-69282848c9b6-istiod-ca-cert" pod "e2284506-257f-4b40-a5c0-69282848c9b6" (UID: "e2284506-257f-4b40-a5c0-69282848c9b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:39.178793 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178549    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-token\" (UniqueName: \"kubernetes.io/projected/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-token\") pod \"cf21565a-4381-4ed0-ad6c-015853c23d1e\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") "
Oct 30 19:20:39.178793 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178704    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf21565a-4381-4ed0-ad6c-015853c23d1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf21565a-4381-4ed0-ad6c-015853c23d1e-istiod-ca-cert\") pod \"cf21565a-4381-4ed0-ad6c-015853c23d1e\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") "
Oct 30 19:20:39.178793 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178776    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-data\") pod \"cf21565a-4381-4ed0-ad6c-015853c23d1e\" (UID: \"cf21565a-4381-4ed0-ad6c-015853c23d1e\") "
Oct 30 19:20:39.179344 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178816    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-istio-data\") pod \"e2284506-257f-4b40-a5c0-69282848c9b6\" (UID: \"e2284506-257f-4b40-a5c0-69282848c9b6\") "
Oct 30 19:20:39.179344 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.178705    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-workload-certs" pod "cf21565a-4381-4ed0-ad6c-015853c23d1e" (UID: "cf21565a-4381-4ed0-ad6c-015853c23d1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.179344 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.179175    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-data" pod "cf21565a-4381-4ed0-ad6c-015853c23d1e" (UID: "cf21565a-4381-4ed0-ad6c-015853c23d1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.179344 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.179211    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf21565a-4381-4ed0-ad6c-015853c23d1e-istiod-ca-cert" pod "cf21565a-4381-4ed0-ad6c-015853c23d1e" (UID: "cf21565a-4381-4ed0-ad6c-015853c23d1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:39.179596 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.179517    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-istio-data" pod "e2284506-257f-4b40-a5c0-69282848c9b6" (UID: "e2284506-257f-4b40-a5c0-69282848c9b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.179944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.179911    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf21565a-4381-4ed0-ad6c-015853c23d1e-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.179944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.179940    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.180097 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.179957    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.180097 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.179973    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.180097 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.179987    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.180097 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.180001    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2284506-257f-4b40-a5c0-69282848c9b6-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.180097 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.180015    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.180097 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.180029    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.180097 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.180043    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.180499 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.180110    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-workload-certs" pod "e2284506-257f-4b40-a5c0-69282848c9b6" (UID: "e2284506-257f-4b40-a5c0-69282848c9b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.181526 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.181500    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2284506-257f-4b40-a5c0-69282848c9b6-istio-token" pod "e2284506-257f-4b40-a5c0-69282848c9b6" (UID: "e2284506-257f-4b40-a5c0-69282848c9b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:39.181617 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.181538    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf21565a-4381-4ed0-ad6c-015853c23d1e-kube-api-access-j85jt" pod "cf21565a-4381-4ed0-ad6c-015853c23d1e" (UID: "cf21565a-4381-4ed0-ad6c-015853c23d1e"). InnerVolumeSpecName "kube-api-access-j85jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:39.181617 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.181560    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-token" pod "cf21565a-4381-4ed0-ad6c-015853c23d1e" (UID: "cf21565a-4381-4ed0-ad6c-015853c23d1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:39.182064 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.182025    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-istio-envoy" pod "e2284506-257f-4b40-a5c0-69282848c9b6" (UID: "e2284506-257f-4b40-a5c0-69282848c9b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.182173 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.182066    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2284506-257f-4b40-a5c0-69282848c9b6-kube-api-access-bp5vc" pod "e2284506-257f-4b40-a5c0-69282848c9b6" (UID: "e2284506-257f-4b40-a5c0-69282848c9b6"). InnerVolumeSpecName "kube-api-access-bp5vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:39.182423 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.182270    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2284506-257f-4b40-a5c0-69282848c9b6-istio-podinfo" pod "e2284506-257f-4b40-a5c0-69282848c9b6" (UID: "e2284506-257f-4b40-a5c0-69282848c9b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:20:39.183036 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.183011    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-podinfo" pod "cf21565a-4381-4ed0-ad6c-015853c23d1e" (UID: "cf21565a-4381-4ed0-ad6c-015853c23d1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:20:39.183709 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.183689    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-envoy" pod "cf21565a-4381-4ed0-ad6c-015853c23d1e" (UID: "cf21565a-4381-4ed0-ad6c-015853c23d1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:39.281856 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.281807    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp5vc\" (UniqueName: \"kubernetes.io/projected/e2284506-257f-4b40-a5c0-69282848c9b6-kube-api-access-bp5vc\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.281856 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.281846    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.281856 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.281856    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2284506-257f-4b40-a5c0-69282848c9b6-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.281856 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.281867    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.281856 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.281876    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2284506-257f-4b40-a5c0-69282848c9b6-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.282321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.281885    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j85jt\" (UniqueName: \"kubernetes.io/projected/cf21565a-4381-4ed0-ad6c-015853c23d1e-kube-api-access-j85jt\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.282321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.281894    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.282321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.281901    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf21565a-4381-4ed0-ad6c-015853c23d1e-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:39.282321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:39.281909    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2284506-257f-4b40-a5c0-69282848c9b6-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:40.075019 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:40.074985    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"
Oct 30 19:20:40.075527 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:40.075021    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr" event={"ID":"cf21565a-4381-4ed0-ad6c-015853c23d1e","Type":"ContainerDied","Data":"d0d8deba71649bc2612d32b2a79afd47d3c8494cae28b9a5b8fb726045ec25b8"}
Oct 30 19:20:40.075527 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:40.075091    4372 scope.go:122] "RemoveContainer" containerID="b8233b629e22056cc8404c2bd383cdef89f87f035f630638b45638ab1495395d"
Oct 30 19:20:40.078004 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:40.077983    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh" event={"ID":"e2284506-257f-4b40-a5c0-69282848c9b6","Type":"ContainerDied","Data":"ced7b7ccaf67bbafeb27bf036451c00b72539deae1d8bb941e64ff1c5d5cae14"}
Oct 30 19:20:40.078109 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:40.078010    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"
Oct 30 19:20:40.090115 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:40.090099    4372 scope.go:122] "RemoveContainer" containerID="1fb60d5766ffae739154e091bbc4a5dee22f04ce5a1512d60187983196eede0c"
Oct 30 19:20:40.104711 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:40.104687    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"]
Oct 30 19:20:40.134738 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:40.112637    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-jhvfr"]
Oct 30 19:20:40.146833 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:40.146813    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"]
Oct 30 19:20:40.152589 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:40.152567    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-g8hdh"]
Oct 30 19:20:40.276429 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:40.276399    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36220b42-54dc-4dd2-aac8-e44b93f61a55" path="/var/lib/kubelet/pods/36220b42-54dc-4dd2-aac8-e44b93f61a55/volumes"
Oct 30 19:20:40.276943 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:40.276928    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf21565a-4381-4ed0-ad6c-015853c23d1e" path="/var/lib/kubelet/pods/cf21565a-4381-4ed0-ad6c-015853c23d1e/volumes"
Oct 30 19:20:40.277554 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:40.277535    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2284506-257f-4b40-a5c0-69282848c9b6" path="/var/lib/kubelet/pods/e2284506-257f-4b40-a5c0-69282848c9b6/volumes"
Oct 30 19:20:41.342225 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:41.342183    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45" podUID="15030cae-5b51-4752-a278-051ab0ca7716" containerName="payload-processing" containerID="cri-o://e77b377ad3a4d16931452f058153afa9f199d00a63451fb58d84ff4a179b82b7" gracePeriod=25
Oct 30 19:20:41.384152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:41.381191    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk" podUID="40b5f840-1e90-4f4c-84de-99857ceac633" containerName="payload-pre-processing" containerID="cri-o://5b1305e24dd833987fd601cd2550e7eb9d1ce367f84dedbac411c26d5c7b1560" gracePeriod=25
Oct 30 19:20:42.253880 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:42.253841    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk" podUID="40b5f840-1e90-4f4c-84de-99857ceac633" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:42.254060 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:42.253897    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45" podUID="15030cae-5b51-4752-a278-051ab0ca7716" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:43.630299 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:43.630250    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"]
Oct 30 19:20:43.630810 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:43.630497    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w" podUID="125aa336-1adb-4d5c-bc7b-42e1b93d85fe" containerName="maas-api" containerID="cri-o://460eb81fc6adf56d9a022b607526696744ed9c81385430f6053622bd4e6da868" gracePeriod=30
Oct 30 19:20:43.833562 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:43.828248    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"]
Oct 30 19:20:43.940575 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:43.940546    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 19:20:44.041121 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.041064    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-maas-api-tls\") pod \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") "
Oct 30 19:20:44.041360 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.041340    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-metrics-certs\") pod \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") "
Oct 30 19:20:44.041449 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.041419    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-kube-api-access-shk9k\" (UniqueName: \"kubernetes.io/projected/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-kube-api-access-shk9k\") pod \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") "
Oct 30 19:20:44.041525 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.041504    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-openshift-service-ca\") pod \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\" (UID: \"125aa336-1adb-4d5c-bc7b-42e1b93d85fe\") "
Oct 30 19:20:44.042345 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.042313    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-openshift-service-ca" pod "125aa336-1adb-4d5c-bc7b-42e1b93d85fe" (UID: "125aa336-1adb-4d5c-bc7b-42e1b93d85fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:44.043823 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.043793    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-maas-api-tls" pod "125aa336-1adb-4d5c-bc7b-42e1b93d85fe" (UID: "125aa336-1adb-4d5c-bc7b-42e1b93d85fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:44.044376 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.043903    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-metrics-certs" pod "125aa336-1adb-4d5c-bc7b-42e1b93d85fe" (UID: "125aa336-1adb-4d5c-bc7b-42e1b93d85fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:44.049978 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.049948    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-kube-api-access-shk9k" pod "125aa336-1adb-4d5c-bc7b-42e1b93d85fe" (UID: "125aa336-1adb-4d5c-bc7b-42e1b93d85fe"). InnerVolumeSpecName "kube-api-access-shk9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:44.111225 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.111197    4372 generic.go:356] "Generic (PLEG): container finished" podID="125aa336-1adb-4d5c-bc7b-42e1b93d85fe" containerID="460eb81fc6adf56d9a022b607526696744ed9c81385430f6053622bd4e6da868" exitCode=0
Oct 30 19:20:44.111334 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.111244    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"
Oct 30 19:20:44.111334 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.111277    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w" event={"ID":"125aa336-1adb-4d5c-bc7b-42e1b93d85fe","Type":"ContainerDied","Data":"460eb81fc6adf56d9a022b607526696744ed9c81385430f6053622bd4e6da868"}
Oct 30 19:20:44.111414 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.111333    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w" event={"ID":"125aa336-1adb-4d5c-bc7b-42e1b93d85fe","Type":"ContainerDied","Data":"a6c2d6d85fdc575c86c88fa63258a0ff764062f41ba5c94c69e3122514738902"}
Oct 30 19:20:44.111414 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.111358    4372 scope.go:122] "RemoveContainer" containerID="460eb81fc6adf56d9a022b607526696744ed9c81385430f6053622bd4e6da868"
Oct 30 19:20:44.121752 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.121733    4372 scope.go:122] "RemoveContainer" containerID="460eb81fc6adf56d9a022b607526696744ed9c81385430f6053622bd4e6da868"
Oct 30 19:20:44.122013 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:20:44.121991    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460eb81fc6adf56d9a022b607526696744ed9c81385430f6053622bd4e6da868\": container with ID starting with 460eb81fc6adf56d9a022b607526696744ed9c81385430f6053622bd4e6da868 not found: ID does not exist" containerID="460eb81fc6adf56d9a022b607526696744ed9c81385430f6053622bd4e6da868"
Oct 30 19:20:44.122078 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.122021    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460eb81fc6adf56d9a022b607526696744ed9c81385430f6053622bd4e6da868"} err="failed to get container status \"460eb81fc6adf56d9a022b607526696744ed9c81385430f6053622bd4e6da868\": rpc error: code = NotFound desc = could not find container \"460eb81fc6adf56d9a022b607526696744ed9c81385430f6053622bd4e6da868\": container with ID starting with 460eb81fc6adf56d9a022b607526696744ed9c81385430f6053622bd4e6da868 not found: ID does not exist"
Oct 30 19:20:44.143164 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.143113    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:44.143245 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.143169    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shk9k\" (UniqueName: \"kubernetes.io/projected/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-kube-api-access-shk9k\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:44.143245 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.143181    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:44.143245 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.143189    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125aa336-1adb-4d5c-bc7b-42e1b93d85fe-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:44.149519 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.149499    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"]
Oct 30 19:20:44.157740 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.157690    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-86c150-6475d8fc76-cqg5w"]
Oct 30 19:20:44.383270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.383230    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="125aa336-1adb-4d5c-bc7b-42e1b93d85fe" path="/var/lib/kubelet/pods/125aa336-1adb-4d5c-bc7b-42e1b93d85fe/volumes"
Oct 30 19:20:44.586796 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.586765    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"]
Oct 30 19:20:44.602894 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:44.601743    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"]
Oct 30 19:20:48.829294 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:48.829213    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg" podUID="3ef41158-9775-4f62-a109-49e3d21649c9" containerName="payload-processing" containerID="cri-o://3b31a0f19ba40a68471d2e737335a1703eaa055581388ec3365da50893fb1728" gracePeriod=25
Oct 30 19:20:49.587730 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:49.587686    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn" podUID="a48daad3-b6ea-4e65-87dc-03a1c811a2cb" containerName="payload-processing" containerID="cri-o://4e800aee892fbcca4a611a51e12f827471d6dba87e178de1629bfda73682c6b8" gracePeriod=25
Oct 30 19:20:49.602200 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:49.602156    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm" podUID="387edd5f-c28a-4732-ad0a-6af167143726" containerName="payload-pre-processing" containerID="cri-o://2ef2bc82034a2e592b05bd1060d996dfdbba9e823d6c0d58f38bd5310e2ea959" gracePeriod=25
Oct 30 19:20:49.830235 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:49.830184    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg" podUID="3ef41158-9775-4f62-a109-49e3d21649c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:52.091876 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:52.091840    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm" podUID="387edd5f-c28a-4732-ad0a-6af167143726" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:52.254242 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:52.254209    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk" podUID="40b5f840-1e90-4f4c-84de-99857ceac633" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:52.254409 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:52.254235    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45" podUID="15030cae-5b51-4752-a278-051ab0ca7716" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:52.873322 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:52.873285    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"]
Oct 30 19:20:52.873877 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:52.873837    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6" podUID="3028a9d2-a829-4420-a287-ce7c2ea64334" containerName="istio-proxy" containerID="cri-o://3cf3d14557f167f0ad1afd6d9ea195cd58cea5f19ada9078b6959c9214bdf7cb" gracePeriod=30
Oct 30 19:20:53.102967 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:53.102699    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn" podUID="a48daad3-b6ea-4e65-87dc-03a1c811a2cb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:57.302656 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:57.302619    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"]
Oct 30 19:20:57.303560 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:57.302882    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs" podUID="dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" containerName="istio-proxy" containerID="cri-o://f717a6ce2d278645d05efe8bc49cf7d3ba901a90eb3ffc4d183fa1c648e93ef0" gracePeriod=30
Oct 30 19:20:57.696180 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:20:57.696107    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:20:58.151359 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.151329    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 19:20:58.155267 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.155240    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-token\" (UniqueName: \"kubernetes.io/projected/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-token\") pod \"3028a9d2-a829-4420-a287-ce7c2ea64334\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") "
Oct 30 19:20:58.155428 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.155412    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-data\") pod \"3028a9d2-a829-4420-a287-ce7c2ea64334\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") "
Oct 30 19:20:58.155501 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.155482    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3028a9d2-a829-4420-a287-ce7c2ea64334-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3028a9d2-a829-4420-a287-ce7c2ea64334-istiod-ca-cert\") pod \"3028a9d2-a829-4420-a287-ce7c2ea64334\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") "
Oct 30 19:20:58.155570 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.155546    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-envoy\") pod \"3028a9d2-a829-4420-a287-ce7c2ea64334\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") "
Oct 30 19:20:58.155628 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.155580    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-credential-socket\") pod \"3028a9d2-a829-4420-a287-ce7c2ea64334\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") "
Oct 30 19:20:58.155686 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.155647    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-workload-certs\") pod \"3028a9d2-a829-4420-a287-ce7c2ea64334\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") "
Oct 30 19:20:58.155748 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.155699    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-data" pod "3028a9d2-a829-4420-a287-ce7c2ea64334" (UID: "3028a9d2-a829-4420-a287-ce7c2ea64334"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:58.155748 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.155717    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3028a9d2-a829-4420-a287-ce7c2ea64334-kube-api-access-lkplh\" (UniqueName: \"kubernetes.io/projected/3028a9d2-a829-4420-a287-ce7c2ea64334-kube-api-access-lkplh\") pod \"3028a9d2-a829-4420-a287-ce7c2ea64334\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") "
Oct 30 19:20:58.155858 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.155792    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-workload-socket\") pod \"3028a9d2-a829-4420-a287-ce7c2ea64334\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") "
Oct 30 19:20:58.155921 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.155858    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-podinfo\") pod \"3028a9d2-a829-4420-a287-ce7c2ea64334\" (UID: \"3028a9d2-a829-4420-a287-ce7c2ea64334\") "
Oct 30 19:20:58.155921 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.155895    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3028a9d2-a829-4420-a287-ce7c2ea64334-istiod-ca-cert" pod "3028a9d2-a829-4420-a287-ce7c2ea64334" (UID: "3028a9d2-a829-4420-a287-ce7c2ea64334"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:58.156172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.156066    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-workload-certs" pod "3028a9d2-a829-4420-a287-ce7c2ea64334" (UID: "3028a9d2-a829-4420-a287-ce7c2ea64334"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:58.156172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.156108    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-credential-socket" pod "3028a9d2-a829-4420-a287-ce7c2ea64334" (UID: "3028a9d2-a829-4420-a287-ce7c2ea64334"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:58.156336 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.156193    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-workload-socket" pod "3028a9d2-a829-4420-a287-ce7c2ea64334" (UID: "3028a9d2-a829-4420-a287-ce7c2ea64334"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:58.156868 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.156803    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.156868 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.156864    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3028a9d2-a829-4420-a287-ce7c2ea64334-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.157025 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.156882    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.157025 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.156896    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.157025 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.156910    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.157946 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.157919    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-token" pod "3028a9d2-a829-4420-a287-ce7c2ea64334" (UID: "3028a9d2-a829-4420-a287-ce7c2ea64334"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:58.158055 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.158030    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-envoy" pod "3028a9d2-a829-4420-a287-ce7c2ea64334" (UID: "3028a9d2-a829-4420-a287-ce7c2ea64334"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:58.158390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.158368    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-podinfo" pod "3028a9d2-a829-4420-a287-ce7c2ea64334" (UID: "3028a9d2-a829-4420-a287-ce7c2ea64334"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:20:58.158489 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.158472    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3028a9d2-a829-4420-a287-ce7c2ea64334-kube-api-access-lkplh" pod "3028a9d2-a829-4420-a287-ce7c2ea64334" (UID: "3028a9d2-a829-4420-a287-ce7c2ea64334"). InnerVolumeSpecName "kube-api-access-lkplh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:58.224895 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.224862    4372 generic.go:356] "Generic (PLEG): container finished" podID="3028a9d2-a829-4420-a287-ce7c2ea64334" containerID="3cf3d14557f167f0ad1afd6d9ea195cd58cea5f19ada9078b6959c9214bdf7cb" exitCode=0
Oct 30 19:20:58.225099 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.224929    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"
Oct 30 19:20:58.225099 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.224950    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6" event={"ID":"3028a9d2-a829-4420-a287-ce7c2ea64334","Type":"ContainerDied","Data":"3cf3d14557f167f0ad1afd6d9ea195cd58cea5f19ada9078b6959c9214bdf7cb"}
Oct 30 19:20:58.225099 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.225000    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6" event={"ID":"3028a9d2-a829-4420-a287-ce7c2ea64334","Type":"ContainerDied","Data":"163989e600716e8938d221aba1f7ecf84b4a07ee517ebbc3923416029a2755cb"}
Oct 30 19:20:58.225099 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.225021    4372 scope.go:122] "RemoveContainer" containerID="3cf3d14557f167f0ad1afd6d9ea195cd58cea5f19ada9078b6959c9214bdf7cb"
Oct 30 19:20:58.236667 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.236647    4372 scope.go:122] "RemoveContainer" containerID="4935fa0b08a5dfb1063a20173ba3824513c0377ebd70401a42d9a3368f6013a8"
Oct 30 19:20:58.249593 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.249571    4372 scope.go:122] "RemoveContainer" containerID="3cf3d14557f167f0ad1afd6d9ea195cd58cea5f19ada9078b6959c9214bdf7cb"
Oct 30 19:20:58.249888 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:20:58.249865    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf3d14557f167f0ad1afd6d9ea195cd58cea5f19ada9078b6959c9214bdf7cb\": container with ID starting with 3cf3d14557f167f0ad1afd6d9ea195cd58cea5f19ada9078b6959c9214bdf7cb not found: ID does not exist" containerID="3cf3d14557f167f0ad1afd6d9ea195cd58cea5f19ada9078b6959c9214bdf7cb"
Oct 30 19:20:58.249967 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.249898    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf3d14557f167f0ad1afd6d9ea195cd58cea5f19ada9078b6959c9214bdf7cb"} err="failed to get container status \"3cf3d14557f167f0ad1afd6d9ea195cd58cea5f19ada9078b6959c9214bdf7cb\": rpc error: code = NotFound desc = could not find container \"3cf3d14557f167f0ad1afd6d9ea195cd58cea5f19ada9078b6959c9214bdf7cb\": container with ID starting with 3cf3d14557f167f0ad1afd6d9ea195cd58cea5f19ada9078b6959c9214bdf7cb not found: ID does not exist"
Oct 30 19:20:58.249967 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.249919    4372 scope.go:122] "RemoveContainer" containerID="4935fa0b08a5dfb1063a20173ba3824513c0377ebd70401a42d9a3368f6013a8"
Oct 30 19:20:58.250255 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:20:58.250233    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4935fa0b08a5dfb1063a20173ba3824513c0377ebd70401a42d9a3368f6013a8\": container with ID starting with 4935fa0b08a5dfb1063a20173ba3824513c0377ebd70401a42d9a3368f6013a8 not found: ID does not exist" containerID="4935fa0b08a5dfb1063a20173ba3824513c0377ebd70401a42d9a3368f6013a8"
Oct 30 19:20:58.250321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.250262    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4935fa0b08a5dfb1063a20173ba3824513c0377ebd70401a42d9a3368f6013a8"} err="failed to get container status \"4935fa0b08a5dfb1063a20173ba3824513c0377ebd70401a42d9a3368f6013a8\": rpc error: code = NotFound desc = could not find container \"4935fa0b08a5dfb1063a20173ba3824513c0377ebd70401a42d9a3368f6013a8\": container with ID starting with 4935fa0b08a5dfb1063a20173ba3824513c0377ebd70401a42d9a3368f6013a8 not found: ID does not exist"
Oct 30 19:20:58.258467 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.258443    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.258555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.258469    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkplh\" (UniqueName: \"kubernetes.io/projected/3028a9d2-a829-4420-a287-ce7c2ea64334-kube-api-access-lkplh\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.258555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.258480    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.258555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.258489    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3028a9d2-a829-4420-a287-ce7c2ea64334-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:58.267387 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.267366    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"]
Oct 30 19:20:58.274158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:58.274116    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-86c150-openshift-default-7d7795f9b9-2rks6"]
Oct 30 19:20:59.829248 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:20:59.829209    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg" podUID="3ef41158-9775-4f62-a109-49e3d21649c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:00.266259 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:00.266174    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3028a9d2-a829-4420-a287-ce7c2ea64334" path="/var/lib/kubelet/pods/3028a9d2-a829-4420-a287-ce7c2ea64334/volumes"
Oct 30 19:21:01.682815 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:01.682793    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"
Oct 30 19:21:01.699853 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:01.699835    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"
Oct 30 19:21:01.715400 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:01.715373    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15030cae-5b51-4752-a278-051ab0ca7716-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15030cae-5b51-4752-a278-051ab0ca7716-plugins-config-volume\") pod \"15030cae-5b51-4752-a278-051ab0ca7716\" (UID: \"15030cae-5b51-4752-a278-051ab0ca7716\") "
Oct 30 19:21:01.715548 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:01.715533    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15030cae-5b51-4752-a278-051ab0ca7716-kube-api-access-ctpxm\" (UniqueName: \"kubernetes.io/projected/15030cae-5b51-4752-a278-051ab0ca7716-kube-api-access-ctpxm\") pod \"15030cae-5b51-4752-a278-051ab0ca7716\" (UID: \"15030cae-5b51-4752-a278-051ab0ca7716\") "
Oct 30 19:21:01.715802 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:01.715778    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15030cae-5b51-4752-a278-051ab0ca7716-plugins-config-volume" pod "15030cae-5b51-4752-a278-051ab0ca7716" (UID: "15030cae-5b51-4752-a278-051ab0ca7716"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:01.716101 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:01.716079    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15030cae-5b51-4752-a278-051ab0ca7716-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:01.717751 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:01.717725    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15030cae-5b51-4752-a278-051ab0ca7716-kube-api-access-ctpxm" pod "15030cae-5b51-4752-a278-051ab0ca7716" (UID: "15030cae-5b51-4752-a278-051ab0ca7716"). InnerVolumeSpecName "kube-api-access-ctpxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:01.816705 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:01.816673    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b5f840-1e90-4f4c-84de-99857ceac633-kube-api-access-jsvck\" (UniqueName: \"kubernetes.io/projected/40b5f840-1e90-4f4c-84de-99857ceac633-kube-api-access-jsvck\") pod \"40b5f840-1e90-4f4c-84de-99857ceac633\" (UID: \"40b5f840-1e90-4f4c-84de-99857ceac633\") "
Oct 30 19:21:01.816896 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:01.816746    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b5f840-1e90-4f4c-84de-99857ceac633-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b5f840-1e90-4f4c-84de-99857ceac633-plugins-config-volume\") pod \"40b5f840-1e90-4f4c-84de-99857ceac633\" (UID: \"40b5f840-1e90-4f4c-84de-99857ceac633\") "
Oct 30 19:21:01.817264 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:01.817227    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b5f840-1e90-4f4c-84de-99857ceac633-plugins-config-volume" pod "40b5f840-1e90-4f4c-84de-99857ceac633" (UID: "40b5f840-1e90-4f4c-84de-99857ceac633"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:01.817448 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:01.817425    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b5f840-1e90-4f4c-84de-99857ceac633-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:01.817532 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:01.817455    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctpxm\" (UniqueName: \"kubernetes.io/projected/15030cae-5b51-4752-a278-051ab0ca7716-kube-api-access-ctpxm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:01.819098 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:01.819075    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b5f840-1e90-4f4c-84de-99857ceac633-kube-api-access-jsvck" pod "40b5f840-1e90-4f4c-84de-99857ceac633" (UID: "40b5f840-1e90-4f4c-84de-99857ceac633"). InnerVolumeSpecName "kube-api-access-jsvck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:01.918798 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:01.918734    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsvck\" (UniqueName: \"kubernetes.io/projected/40b5f840-1e90-4f4c-84de-99857ceac633-kube-api-access-jsvck\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.068268 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.068232    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"]
Oct 30 19:21:02.068492 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.068465    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb" podUID="b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" containerName="istio-proxy" containerID="cri-o://52ecf7652f868d7d9cde9f9a039d49ed1c9f562cf2395ac24c3b45a7c09cb153" gracePeriod=30
Oct 30 19:21:02.091916 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.091884    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm" podUID="387edd5f-c28a-4732-ad0a-6af167143726" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:02.312204 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.311590    4372 generic.go:356] "Generic (PLEG): container finished" podID="40b5f840-1e90-4f4c-84de-99857ceac633" containerID="5b1305e24dd833987fd601cd2550e7eb9d1ce367f84dedbac411c26d5c7b1560" exitCode=0
Oct 30 19:21:02.312204 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.311744    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"
Oct 30 19:21:02.328152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.326291    4372 generic.go:356] "Generic (PLEG): container finished" podID="15030cae-5b51-4752-a278-051ab0ca7716" containerID="e77b377ad3a4d16931452f058153afa9f199d00a63451fb58d84ff4a179b82b7" exitCode=0
Oct 30 19:21:02.328152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.326555    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"
Oct 30 19:21:02.355174 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.350032    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk" event={"ID":"40b5f840-1e90-4f4c-84de-99857ceac633","Type":"ContainerDied","Data":"5b1305e24dd833987fd601cd2550e7eb9d1ce367f84dedbac411c26d5c7b1560"}
Oct 30 19:21:02.355174 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.350102    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk" event={"ID":"40b5f840-1e90-4f4c-84de-99857ceac633","Type":"ContainerDied","Data":"7f3d49ffb7c4fd7e90e514b508365dbea2c650318e17e5da1bf9df28a7788ef8"}
Oct 30 19:21:02.355174 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.350147    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45" event={"ID":"15030cae-5b51-4752-a278-051ab0ca7716","Type":"ContainerDied","Data":"e77b377ad3a4d16931452f058153afa9f199d00a63451fb58d84ff4a179b82b7"}
Oct 30 19:21:02.355174 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.350174    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45" event={"ID":"15030cae-5b51-4752-a278-051ab0ca7716","Type":"ContainerDied","Data":"144eb77eb8047a9ae9e8dbbc19cc6045df9d947dce976285e5bf1274473f9df9"}
Oct 30 19:21:02.355174 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.350202    4372 scope.go:122] "RemoveContainer" containerID="5b1305e24dd833987fd601cd2550e7eb9d1ce367f84dedbac411c26d5c7b1560"
Oct 30 19:21:02.417276 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.417245    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"]
Oct 30 19:21:02.418724 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.418703    4372 scope.go:122] "RemoveContainer" containerID="5b1305e24dd833987fd601cd2550e7eb9d1ce367f84dedbac411c26d5c7b1560"
Oct 30 19:21:02.419064 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:02.419041    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b1305e24dd833987fd601cd2550e7eb9d1ce367f84dedbac411c26d5c7b1560\": container with ID starting with 5b1305e24dd833987fd601cd2550e7eb9d1ce367f84dedbac411c26d5c7b1560 not found: ID does not exist" containerID="5b1305e24dd833987fd601cd2550e7eb9d1ce367f84dedbac411c26d5c7b1560"
Oct 30 19:21:02.419165 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.419073    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b1305e24dd833987fd601cd2550e7eb9d1ce367f84dedbac411c26d5c7b1560"} err="failed to get container status \"5b1305e24dd833987fd601cd2550e7eb9d1ce367f84dedbac411c26d5c7b1560\": rpc error: code = NotFound desc = could not find container \"5b1305e24dd833987fd601cd2550e7eb9d1ce367f84dedbac411c26d5c7b1560\": container with ID starting with 5b1305e24dd833987fd601cd2550e7eb9d1ce367f84dedbac411c26d5c7b1560 not found: ID does not exist"
Oct 30 19:21:02.419165 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.419095    4372 scope.go:122] "RemoveContainer" containerID="e77b377ad3a4d16931452f058153afa9f199d00a63451fb58d84ff4a179b82b7"
Oct 30 19:21:02.422158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.422117    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-86c150-5784f9b7f9-7vwzk"]
Oct 30 19:21:02.429870 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.429840    4372 scope.go:122] "RemoveContainer" containerID="e77b377ad3a4d16931452f058153afa9f199d00a63451fb58d84ff4a179b82b7"
Oct 30 19:21:02.430225 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:02.430203    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e77b377ad3a4d16931452f058153afa9f199d00a63451fb58d84ff4a179b82b7\": container with ID starting with e77b377ad3a4d16931452f058153afa9f199d00a63451fb58d84ff4a179b82b7 not found: ID does not exist" containerID="e77b377ad3a4d16931452f058153afa9f199d00a63451fb58d84ff4a179b82b7"
Oct 30 19:21:02.430277 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.430233    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e77b377ad3a4d16931452f058153afa9f199d00a63451fb58d84ff4a179b82b7"} err="failed to get container status \"e77b377ad3a4d16931452f058153afa9f199d00a63451fb58d84ff4a179b82b7\": rpc error: code = NotFound desc = could not find container \"e77b377ad3a4d16931452f058153afa9f199d00a63451fb58d84ff4a179b82b7\": container with ID starting with e77b377ad3a4d16931452f058153afa9f199d00a63451fb58d84ff4a179b82b7 not found: ID does not exist"
Oct 30 19:21:02.438709 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.438682    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"]
Oct 30 19:21:02.443376 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.443346    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-86c150-678c8d8798-wgd45"]
Oct 30 19:21:02.612387 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.612364    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:21:02.745222 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.745187    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-credential-socket\") pod \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") "
Oct 30 19:21:02.745732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.745253    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istiod-ca-cert\") pod \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") "
Oct 30 19:21:02.745732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.745393    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-kube-api-access-k24j9\" (UniqueName: \"kubernetes.io/projected/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-kube-api-access-k24j9\") pod \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") "
Oct 30 19:21:02.745732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.745422    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-token\" (UniqueName: \"kubernetes.io/projected/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-token\") pod \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") "
Oct 30 19:21:02.745732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.745465    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-credential-socket" pod "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" (UID: "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:02.745732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.745502    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-data\") pod \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") "
Oct 30 19:21:02.745732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.745598    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-podinfo\") pod \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") "
Oct 30 19:21:02.745732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.745731    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-envoy\") pod \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") "
Oct 30 19:21:02.746194 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.745764    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-workload-socket\") pod \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") "
Oct 30 19:21:02.746194 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.745831    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-workload-certs\") pod \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\" (UID: \"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4\") "
Oct 30 19:21:02.746667 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.746422    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istiod-ca-cert" pod "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" (UID: "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:02.746667 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.746628    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-workload-certs" pod "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" (UID: "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:02.746667 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.746635    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.746667 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.746656    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.748055 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.746663    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-data" pod "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" (UID: "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:02.748055 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.746730    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-workload-socket" pod "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" (UID: "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:02.751459 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.751420    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-token" pod "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" (UID: "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:02.752503 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.752458    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-envoy" pod "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" (UID: "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:02.759546 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.759520    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-podinfo" pod "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" (UID: "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:21:02.759637 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.759623    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-kube-api-access-k24j9" pod "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" (UID: "dac6b8fc-c95d-43a7-b18d-2c4953b9cda4"). InnerVolumeSpecName "kube-api-access-k24j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:02.847889 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.847835    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.847889 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.847861    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.847889 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.847870    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.847889 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.847879    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.847889 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.847890    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k24j9\" (UniqueName: \"kubernetes.io/projected/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-kube-api-access-k24j9\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.848114 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.847899    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.848114 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:02.847907    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:03.106888 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:03.106774    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn" podUID="a48daad3-b6ea-4e65-87dc-03a1c811a2cb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:03.334975 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:03.334938    4372 generic.go:356] "Generic (PLEG): container finished" podID="dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" containerID="f717a6ce2d278645d05efe8bc49cf7d3ba901a90eb3ffc4d183fa1c648e93ef0" exitCode=0
Oct 30 19:21:03.335186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:03.334997    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"
Oct 30 19:21:03.335186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:03.335031    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs" event={"ID":"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4","Type":"ContainerDied","Data":"f717a6ce2d278645d05efe8bc49cf7d3ba901a90eb3ffc4d183fa1c648e93ef0"}
Oct 30 19:21:03.335186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:03.335090    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs" event={"ID":"dac6b8fc-c95d-43a7-b18d-2c4953b9cda4","Type":"ContainerDied","Data":"3991e712e45248d097092a192cc4c27997103facc250c3f3b3f5da021a012d40"}
Oct 30 19:21:03.335186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:03.335114    4372 scope.go:122] "RemoveContainer" containerID="f717a6ce2d278645d05efe8bc49cf7d3ba901a90eb3ffc4d183fa1c648e93ef0"
Oct 30 19:21:03.346829 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:03.346807    4372 scope.go:122] "RemoveContainer" containerID="f717a6ce2d278645d05efe8bc49cf7d3ba901a90eb3ffc4d183fa1c648e93ef0"
Oct 30 19:21:03.347097 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:03.347076    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f717a6ce2d278645d05efe8bc49cf7d3ba901a90eb3ffc4d183fa1c648e93ef0\": container with ID starting with f717a6ce2d278645d05efe8bc49cf7d3ba901a90eb3ffc4d183fa1c648e93ef0 not found: ID does not exist" containerID="f717a6ce2d278645d05efe8bc49cf7d3ba901a90eb3ffc4d183fa1c648e93ef0"
Oct 30 19:21:03.347178 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:03.347106    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f717a6ce2d278645d05efe8bc49cf7d3ba901a90eb3ffc4d183fa1c648e93ef0"} err="failed to get container status \"f717a6ce2d278645d05efe8bc49cf7d3ba901a90eb3ffc4d183fa1c648e93ef0\": rpc error: code = NotFound desc = could not find container \"f717a6ce2d278645d05efe8bc49cf7d3ba901a90eb3ffc4d183fa1c648e93ef0\": container with ID starting with f717a6ce2d278645d05efe8bc49cf7d3ba901a90eb3ffc4d183fa1c648e93ef0 not found: ID does not exist"
Oct 30 19:21:03.362115 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:03.362066    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"]
Oct 30 19:21:03.374736 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:03.374714    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-hkphs"]
Oct 30 19:21:04.267403 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:04.267372    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15030cae-5b51-4752-a278-051ab0ca7716" path="/var/lib/kubelet/pods/15030cae-5b51-4752-a278-051ab0ca7716/volumes"
Oct 30 19:21:04.267819 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:04.267802    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b5f840-1e90-4f4c-84de-99857ceac633" path="/var/lib/kubelet/pods/40b5f840-1e90-4f4c-84de-99857ceac633/volumes"
Oct 30 19:21:04.268292 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:04.268276    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" path="/var/lib/kubelet/pods/dac6b8fc-c95d-43a7-b18d-2c4953b9cda4/volumes"
Oct 30 19:21:07.343774 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.343749    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:21:07.368023 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.367956    4372 generic.go:356] "Generic (PLEG): container finished" podID="b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" containerID="52ecf7652f868d7d9cde9f9a039d49ed1c9f562cf2395ac24c3b45a7c09cb153" exitCode=0
Oct 30 19:21:07.368149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.368072    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"
Oct 30 19:21:07.368149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.368103    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb" event={"ID":"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4","Type":"ContainerDied","Data":"52ecf7652f868d7d9cde9f9a039d49ed1c9f562cf2395ac24c3b45a7c09cb153"}
Oct 30 19:21:07.368229 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.368201    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb" event={"ID":"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4","Type":"ContainerDied","Data":"0b37fc37cc8f90e1022b78aa7d3f6414d48394e9f4b0370e1e5178d471e24e91"}
Oct 30 19:21:07.368229 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.368226    4372 scope.go:122] "RemoveContainer" containerID="52ecf7652f868d7d9cde9f9a039d49ed1c9f562cf2395ac24c3b45a7c09cb153"
Oct 30 19:21:07.378453 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.378435    4372 scope.go:122] "RemoveContainer" containerID="52ecf7652f868d7d9cde9f9a039d49ed1c9f562cf2395ac24c3b45a7c09cb153"
Oct 30 19:21:07.378742 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:07.378719    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ecf7652f868d7d9cde9f9a039d49ed1c9f562cf2395ac24c3b45a7c09cb153\": container with ID starting with 52ecf7652f868d7d9cde9f9a039d49ed1c9f562cf2395ac24c3b45a7c09cb153 not found: ID does not exist" containerID="52ecf7652f868d7d9cde9f9a039d49ed1c9f562cf2395ac24c3b45a7c09cb153"
Oct 30 19:21:07.378814 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.378753    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ecf7652f868d7d9cde9f9a039d49ed1c9f562cf2395ac24c3b45a7c09cb153"} err="failed to get container status \"52ecf7652f868d7d9cde9f9a039d49ed1c9f562cf2395ac24c3b45a7c09cb153\": rpc error: code = NotFound desc = could not find container \"52ecf7652f868d7d9cde9f9a039d49ed1c9f562cf2395ac24c3b45a7c09cb153\": container with ID starting with 52ecf7652f868d7d9cde9f9a039d49ed1c9f562cf2395ac24c3b45a7c09cb153 not found: ID does not exist"
Oct 30 19:21:07.425321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.425298    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-workload-socket\") pod \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") "
Oct 30 19:21:07.425460 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.425338    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-podinfo\") pod \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") "
Oct 30 19:21:07.425460 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.425399    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-kube-api-access-r25tq\" (UniqueName: \"kubernetes.io/projected/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-kube-api-access-r25tq\") pod \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") "
Oct 30 19:21:07.425597 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.425508    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-envoy\") pod \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") "
Oct 30 19:21:07.425597 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.425585    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-credential-socket\") pod \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") "
Oct 30 19:21:07.425721 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.425598    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-workload-socket" pod "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" (UID: "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:07.425721 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.425623    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istiod-ca-cert\") pod \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") "
Oct 30 19:21:07.425834 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.425745    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-token\" (UniqueName: \"kubernetes.io/projected/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-token\") pod \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") "
Oct 30 19:21:07.425834 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.425790    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-workload-certs\") pod \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") "
Oct 30 19:21:07.425948 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.425859    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-data\") pod \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\" (UID: \"b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4\") "
Oct 30 19:21:07.426778 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.426749    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.426941 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.426822    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istiod-ca-cert" pod "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" (UID: "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:07.426941 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.426825    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-credential-socket" pod "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" (UID: "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:07.427189 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.426990    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-data" pod "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" (UID: "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:07.427189 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.427003    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-workload-certs" pod "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" (UID: "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:07.443669 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.443639    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-token" pod "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" (UID: "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:07.443842 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.443766    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-envoy" pod "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" (UID: "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:07.444246 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.444168    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-kube-api-access-r25tq" pod "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" (UID: "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4"). InnerVolumeSpecName "kube-api-access-r25tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:07.446294 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.446257    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-podinfo" pod "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" (UID: "b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:21:07.528534 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.528508    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.528534 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.528535    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r25tq\" (UniqueName: \"kubernetes.io/projected/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-kube-api-access-r25tq\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.528673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.528545    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.528673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.528555    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.528673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.528563    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.528673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.528571    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.528673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.528580    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.528673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.528588    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:07.639927 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.639858    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"]
Oct 30 19:21:07.702309 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.702280    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"]
Oct 30 19:21:07.710431 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:07.710402    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-lsfnb"]
Oct 30 19:21:08.266409 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.266370    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" path="/var/lib/kubelet/pods/b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4/volumes"
Oct 30 19:21:08.695084 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695056    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"]
Oct 30 19:21:08.695695 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695680    4372 cpu_manager.go:414] "containerMap: removing container" podUID="3028a9d2-a829-4420-a287-ce7c2ea64334" containerName="istio-proxy"
Oct 30 19:21:08.695743 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695696    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3028a9d2-a829-4420-a287-ce7c2ea64334" containerName="istio-proxy"
Oct 30 19:21:08.695743 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695711    4372 cpu_manager.go:414] "containerMap: removing container" podUID="36220b42-54dc-4dd2-aac8-e44b93f61a55" containerName="istio-proxy"
Oct 30 19:21:08.695743 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695717    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36220b42-54dc-4dd2-aac8-e44b93f61a55" containerName="istio-proxy"
Oct 30 19:21:08.695743 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695725    4372 cpu_manager.go:414] "containerMap: removing container" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerName="istio-proxy"
Oct 30 19:21:08.695743 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695729    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerName="istio-proxy"
Oct 30 19:21:08.695743 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695736    4372 cpu_manager.go:414] "containerMap: removing container" podUID="fbcd714e-b581-4142-ae56-05bf2a33d52b" containerName="istio-proxy"
Oct 30 19:21:08.695743 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695741    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbcd714e-b581-4142-ae56-05bf2a33d52b" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695753    4372 cpu_manager.go:414] "containerMap: removing container" podUID="cf21565a-4381-4ed0-ad6c-015853c23d1e" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695757    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf21565a-4381-4ed0-ad6c-015853c23d1e" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695766    4372 cpu_manager.go:414] "containerMap: removing container" podUID="dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695770    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695777    4372 cpu_manager.go:414] "containerMap: removing container" podUID="3028a9d2-a829-4420-a287-ce7c2ea64334" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695781    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3028a9d2-a829-4420-a287-ce7c2ea64334" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695790    4372 cpu_manager.go:414] "containerMap: removing container" podUID="4f57597b-6f1c-47e2-9891-5432b04e8fb1" containerName="payload-pre-processing"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695794    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f57597b-6f1c-47e2-9891-5432b04e8fb1" containerName="payload-pre-processing"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695802    4372 cpu_manager.go:414] "containerMap: removing container" podUID="6faafbec-abd9-4f16-a979-e607b39565ee" containerName="payload-processing"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695806    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6faafbec-abd9-4f16-a979-e607b39565ee" containerName="payload-processing"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695815    4372 cpu_manager.go:414] "containerMap: removing container" podUID="36220b42-54dc-4dd2-aac8-e44b93f61a55" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695820    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36220b42-54dc-4dd2-aac8-e44b93f61a55" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695829    4372 cpu_manager.go:414] "containerMap: removing container" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695834    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695841    4372 cpu_manager.go:414] "containerMap: removing container" podUID="40b5f840-1e90-4f4c-84de-99857ceac633" containerName="payload-pre-processing"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695845    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b5f840-1e90-4f4c-84de-99857ceac633" containerName="payload-pre-processing"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695852    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f681d0ac-6413-4ccb-88ba-3e0e278bd663" containerName="payload-processing"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695857    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f681d0ac-6413-4ccb-88ba-3e0e278bd663" containerName="payload-processing"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695863    4372 cpu_manager.go:414] "containerMap: removing container" podUID="6abe5746-4718-4d28-88dc-5b9a54999869" containerName="payload-processing"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695868    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abe5746-4718-4d28-88dc-5b9a54999869" containerName="payload-processing"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695879    4372 cpu_manager.go:414] "containerMap: removing container" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695884    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695893    4372 cpu_manager.go:414] "containerMap: removing container" podUID="e2284506-257f-4b40-a5c0-69282848c9b6" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695897    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2284506-257f-4b40-a5c0-69282848c9b6" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695905    4372 cpu_manager.go:414] "containerMap: removing container" podUID="15030cae-5b51-4752-a278-051ab0ca7716" containerName="payload-processing"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695909    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15030cae-5b51-4752-a278-051ab0ca7716" containerName="payload-processing"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695917    4372 cpu_manager.go:414] "containerMap: removing container" podUID="e09ce3e3-0b45-4f34-926f-a6a067ef44a4" containerName="payload-processing"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695921    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09ce3e3-0b45-4f34-926f-a6a067ef44a4" containerName="payload-processing"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695930    4372 cpu_manager.go:414] "containerMap: removing container" podUID="2e3a7f70-606a-40b1-b541-e279a8438a32" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695935    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3a7f70-606a-40b1-b541-e279a8438a32" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695949    4372 cpu_manager.go:414] "containerMap: removing container" podUID="125aa336-1adb-4d5c-bc7b-42e1b93d85fe" containerName="maas-api"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695954    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125aa336-1adb-4d5c-bc7b-42e1b93d85fe" containerName="maas-api"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695963    4372 cpu_manager.go:414] "containerMap: removing container" podUID="b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695967    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" containerName="istio-proxy"
Oct 30 19:21:08.695962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695975    4372 cpu_manager.go:414] "containerMap: removing container" podUID="9c89796b-9ed3-4d26-a3e1-89525585be35" containerName="payload-processing"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.695980    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c89796b-9ed3-4d26-a3e1-89525585be35" containerName="payload-processing"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696071    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d1c3fc-26d4-40ea-ab5a-9d2758bedbd4" containerName="istio-proxy"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696082    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c89796b-9ed3-4d26-a3e1-89525585be35" containerName="payload-processing"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696088    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="3028a9d2-a829-4420-a287-ce7c2ea64334" containerName="istio-proxy"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696094    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="36220b42-54dc-4dd2-aac8-e44b93f61a55" containerName="istio-proxy"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696100    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerName="istio-proxy"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696106    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbcd714e-b581-4142-ae56-05bf2a33d52b" containerName="istio-proxy"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696114    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf21565a-4381-4ed0-ad6c-015853c23d1e" containerName="istio-proxy"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696120    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac6b8fc-c95d-43a7-b18d-2c4953b9cda4" containerName="istio-proxy"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696146    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="3028a9d2-a829-4420-a287-ce7c2ea64334" containerName="istio-proxy"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696151    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerName="istio-proxy"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696158    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f57597b-6f1c-47e2-9891-5432b04e8fb1" containerName="payload-pre-processing"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696164    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="6faafbec-abd9-4f16-a979-e607b39565ee" containerName="payload-processing"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696171    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="36220b42-54dc-4dd2-aac8-e44b93f61a55" containerName="istio-proxy"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696177    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerName="istio-proxy"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696183    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b5f840-1e90-4f4c-84de-99857ceac633" containerName="payload-pre-processing"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696188    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="f681d0ac-6413-4ccb-88ba-3e0e278bd663" containerName="payload-processing"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696193    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abe5746-4718-4d28-88dc-5b9a54999869" containerName="payload-processing"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696200    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="9313e8c4-ec9e-4735-b58a-ce18fa0a692b" containerName="istio-proxy"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696206    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2284506-257f-4b40-a5c0-69282848c9b6" containerName="istio-proxy"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696211    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="15030cae-5b51-4752-a278-051ab0ca7716" containerName="payload-processing"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696216    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09ce3e3-0b45-4f34-926f-a6a067ef44a4" containerName="payload-processing"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696222    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3a7f70-606a-40b1-b541-e279a8438a32" containerName="istio-proxy"
Oct 30 19:21:08.697075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.696228    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="125aa336-1adb-4d5c-bc7b-42e1b93d85fe" containerName="maas-api"
Oct 30 19:21:08.701153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.701118    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:08.712354 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.712329    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"payload-processing-external-model-plugins-e2e-extmod-a-21d896\""
Oct 30 19:21:08.712354 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.712344    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"openshift-service-ca.crt\""
Oct 30 19:21:08.713342 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.713106    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"]
Oct 30 19:21:08.713868 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.713851    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"kube-root-ca.crt\""
Oct 30 19:21:08.714078 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.714062    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"agc-cred-a05ff35a6b8d\""
Oct 30 19:21:08.714172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.714073    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"routing-overlay\""
Oct 30 19:21:08.714172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.714098    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"payload-processing-external-model-e2e--3f19eb37-dockercfg-phgbc\""
Oct 30 19:21:08.714172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.714099    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"agc-cred-1b4803c0417a\""
Oct 30 19:21:08.714172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.714160    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"agc-cred-676f802b0d13\""
Oct 30 19:21:08.868153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.861754    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65vn2\" (UniqueName: \"kubernetes.io/projected/869ccb82-8beb-464c-9032-9540154538f8-kube-api-access-65vn2\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:08.868153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.861862    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/869ccb82-8beb-464c-9032-9540154538f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:08.868153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.862039    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/869ccb82-8beb-464c-9032-9540154538f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:08.868153 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.862253    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/869ccb82-8beb-464c-9032-9540154538f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:08.963963 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.963845    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/869ccb82-8beb-464c-9032-9540154538f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:08.964148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.963995    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/869ccb82-8beb-464c-9032-9540154538f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:08.964148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.964075    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65vn2\" (UniqueName: \"kubernetes.io/projected/869ccb82-8beb-464c-9032-9540154538f8-kube-api-access-65vn2\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:08.964249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.964153    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/869ccb82-8beb-464c-9032-9540154538f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:08.965460 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.965325    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/869ccb82-8beb-464c-9032-9540154538f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:08.966075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.966047    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/869ccb82-8beb-464c-9032-9540154538f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:08.968371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.968335    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/869ccb82-8beb-464c-9032-9540154538f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:08.978084 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:08.978056    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65vn2\" (UniqueName: \"kubernetes.io/projected/869ccb82-8beb-464c-9032-9540154538f8-kube-api-access-65vn2\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:09.025241 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.025216    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:09.038013 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.037988    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"
Oct 30 19:21:09.069193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.066951    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"]
Oct 30 19:21:09.069193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.067669    4372 cpu_manager.go:414] "containerMap: removing container" podUID="3ef41158-9775-4f62-a109-49e3d21649c9" containerName="payload-processing"
Oct 30 19:21:09.069193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.067688    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef41158-9775-4f62-a109-49e3d21649c9" containerName="payload-processing"
Oct 30 19:21:09.069193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.067700    4372 cpu_manager.go:414] "containerMap: removing container" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerName="istio-proxy"
Oct 30 19:21:09.069193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.067706    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b96f6e-b871-499b-ab8f-18fffaa3ece2" containerName="istio-proxy"
Oct 30 19:21:09.069193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.067827    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef41158-9775-4f62-a109-49e3d21649c9" containerName="payload-processing"
Oct 30 19:21:09.071673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.071650    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:09.081782 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.081313    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d72beb\"/\"agc-cred-14c07f8fe136\""
Oct 30 19:21:09.088468 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.088429    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"]
Oct 30 19:21:09.166287 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.166258    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef41158-9775-4f62-a109-49e3d21649c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef41158-9775-4f62-a109-49e3d21649c9-plugins-config-volume\") pod \"3ef41158-9775-4f62-a109-49e3d21649c9\" (UID: \"3ef41158-9775-4f62-a109-49e3d21649c9\") "
Oct 30 19:21:09.166640 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.166617    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef41158-9775-4f62-a109-49e3d21649c9-kube-api-access-lnl44\" (UniqueName: \"kubernetes.io/projected/3ef41158-9775-4f62-a109-49e3d21649c9-kube-api-access-lnl44\") pod \"3ef41158-9775-4f62-a109-49e3d21649c9\" (UID: \"3ef41158-9775-4f62-a109-49e3d21649c9\") "
Oct 30 19:21:09.166799 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.166619    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef41158-9775-4f62-a109-49e3d21649c9-plugins-config-volume" pod "3ef41158-9775-4f62-a109-49e3d21649c9" (UID: "3ef41158-9775-4f62-a109-49e3d21649c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:09.166940 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.166915    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4934163b-b3eb-4f34-acd6-39a847daa0a8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:09.167007 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.166966    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4934163b-b3eb-4f34-acd6-39a847daa0a8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:09.167212 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.167191    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t48vw\" (UniqueName: \"kubernetes.io/projected/4934163b-b3eb-4f34-acd6-39a847daa0a8-kube-api-access-t48vw\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:09.167310 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.167289    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4934163b-b3eb-4f34-acd6-39a847daa0a8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:09.167515 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.167500    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef41158-9775-4f62-a109-49e3d21649c9-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:09.169467 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.169440    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef41158-9775-4f62-a109-49e3d21649c9-kube-api-access-lnl44" pod "3ef41158-9775-4f62-a109-49e3d21649c9" (UID: "3ef41158-9775-4f62-a109-49e3d21649c9"). InnerVolumeSpecName "kube-api-access-lnl44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:09.268921 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.268852    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4934163b-b3eb-4f34-acd6-39a847daa0a8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:09.268921 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.268894    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4934163b-b3eb-4f34-acd6-39a847daa0a8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:09.269073 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.268951    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t48vw\" (UniqueName: \"kubernetes.io/projected/4934163b-b3eb-4f34-acd6-39a847daa0a8-kube-api-access-t48vw\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:09.269073 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.269019    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4934163b-b3eb-4f34-acd6-39a847daa0a8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:09.269249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.269074    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnl44\" (UniqueName: \"kubernetes.io/projected/3ef41158-9775-4f62-a109-49e3d21649c9-kube-api-access-lnl44\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:09.269797 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.269774    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4934163b-b3eb-4f34-acd6-39a847daa0a8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:09.269867 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.269854    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4934163b-b3eb-4f34-acd6-39a847daa0a8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:09.271521 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.271500    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4934163b-b3eb-4f34-acd6-39a847daa0a8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:09.281978 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.281956    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t48vw\" (UniqueName: \"kubernetes.io/projected/4934163b-b3eb-4f34-acd6-39a847daa0a8-kube-api-access-t48vw\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:09.317990 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:21:09.317967    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869ccb82_8beb_464c_9032_9540154538f8.slice/crio-835aea4dbf712f7e2b0ce2e9e6bf2ef5a977f4d37dacc4992c827fdbaee55767 WatchSource:0}: Error finding container 835aea4dbf712f7e2b0ce2e9e6bf2ef5a977f4d37dacc4992c827fdbaee55767: Status 404 returned error can't find the container with id 835aea4dbf712f7e2b0ce2e9e6bf2ef5a977f4d37dacc4992c827fdbaee55767
Oct 30 19:21:09.318118 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.318075    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"]
Oct 30 19:21:09.384758 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.384722    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl" event={"ID":"869ccb82-8beb-464c-9032-9540154538f8","Type":"ContainerStarted","Data":"065c94441bf8234e9527459961f944f8419a7d5e180abd0f0f35514fdd1219d3"}
Oct 30 19:21:09.384888 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.384777    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl" event={"ID":"869ccb82-8beb-464c-9032-9540154538f8","Type":"ContainerStarted","Data":"835aea4dbf712f7e2b0ce2e9e6bf2ef5a977f4d37dacc4992c827fdbaee55767"}
Oct 30 19:21:09.386386 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.386362    4372 generic.go:356] "Generic (PLEG): container finished" podID="3ef41158-9775-4f62-a109-49e3d21649c9" containerID="3b31a0f19ba40a68471d2e737335a1703eaa055581388ec3365da50893fb1728" exitCode=0
Oct 30 19:21:09.386496 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.386410    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"
Oct 30 19:21:09.386496 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.386432    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg" event={"ID":"3ef41158-9775-4f62-a109-49e3d21649c9","Type":"ContainerDied","Data":"3b31a0f19ba40a68471d2e737335a1703eaa055581388ec3365da50893fb1728"}
Oct 30 19:21:09.386496 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.386481    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg" event={"ID":"3ef41158-9775-4f62-a109-49e3d21649c9","Type":"ContainerDied","Data":"67bb8993a023c6b51a7dc04dfc6875a7b5e6525c599f5e6b940e678e0b9ccda0"}
Oct 30 19:21:09.386649 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.386509    4372 scope.go:122] "RemoveContainer" containerID="3b31a0f19ba40a68471d2e737335a1703eaa055581388ec3365da50893fb1728"
Oct 30 19:21:09.397688 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.397667    4372 scope.go:122] "RemoveContainer" containerID="3b31a0f19ba40a68471d2e737335a1703eaa055581388ec3365da50893fb1728"
Oct 30 19:21:09.397980 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:09.397959    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b31a0f19ba40a68471d2e737335a1703eaa055581388ec3365da50893fb1728\": container with ID starting with 3b31a0f19ba40a68471d2e737335a1703eaa055581388ec3365da50893fb1728 not found: ID does not exist" containerID="3b31a0f19ba40a68471d2e737335a1703eaa055581388ec3365da50893fb1728"
Oct 30 19:21:09.398072 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.397990    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b31a0f19ba40a68471d2e737335a1703eaa055581388ec3365da50893fb1728"} err="failed to get container status \"3b31a0f19ba40a68471d2e737335a1703eaa055581388ec3365da50893fb1728\": rpc error: code = NotFound desc = could not find container \"3b31a0f19ba40a68471d2e737335a1703eaa055581388ec3365da50893fb1728\": container with ID starting with 3b31a0f19ba40a68471d2e737335a1703eaa055581388ec3365da50893fb1728 not found: ID does not exist"
Oct 30 19:21:09.401690 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.401670    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:09.412070 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.412047    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"]
Oct 30 19:21:09.418543 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.418520    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-784fbf-7468d58bb6-6p8qg"]
Oct 30 19:21:09.687388 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.687362    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"]
Oct 30 19:21:09.960388 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.960359    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"
Oct 30 19:21:09.983439 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.983402    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48daad3-b6ea-4e65-87dc-03a1c811a2cb-kube-api-access-brmrk\" (UniqueName: \"kubernetes.io/projected/a48daad3-b6ea-4e65-87dc-03a1c811a2cb-kube-api-access-brmrk\") pod \"a48daad3-b6ea-4e65-87dc-03a1c811a2cb\" (UID: \"a48daad3-b6ea-4e65-87dc-03a1c811a2cb\") "
Oct 30 19:21:09.983595 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.983576    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48daad3-b6ea-4e65-87dc-03a1c811a2cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48daad3-b6ea-4e65-87dc-03a1c811a2cb-plugins-config-volume\") pod \"a48daad3-b6ea-4e65-87dc-03a1c811a2cb\" (UID: \"a48daad3-b6ea-4e65-87dc-03a1c811a2cb\") "
Oct 30 19:21:09.984243 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.984215    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48daad3-b6ea-4e65-87dc-03a1c811a2cb-plugins-config-volume" pod "a48daad3-b6ea-4e65-87dc-03a1c811a2cb" (UID: "a48daad3-b6ea-4e65-87dc-03a1c811a2cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:09.986066 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:09.986043    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48daad3-b6ea-4e65-87dc-03a1c811a2cb-kube-api-access-brmrk" pod "a48daad3-b6ea-4e65-87dc-03a1c811a2cb" (UID: "a48daad3-b6ea-4e65-87dc-03a1c811a2cb"). InnerVolumeSpecName "kube-api-access-brmrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:10.001225 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.001196    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"
Oct 30 19:21:10.084976 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.084944    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387edd5f-c28a-4732-ad0a-6af167143726-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387edd5f-c28a-4732-ad0a-6af167143726-plugins-config-volume\") pod \"387edd5f-c28a-4732-ad0a-6af167143726\" (UID: \"387edd5f-c28a-4732-ad0a-6af167143726\") "
Oct 30 19:21:10.085101 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.085012    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387edd5f-c28a-4732-ad0a-6af167143726-kube-api-access-2cj8c\" (UniqueName: \"kubernetes.io/projected/387edd5f-c28a-4732-ad0a-6af167143726-kube-api-access-2cj8c\") pod \"387edd5f-c28a-4732-ad0a-6af167143726\" (UID: \"387edd5f-c28a-4732-ad0a-6af167143726\") "
Oct 30 19:21:10.085401 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.085378    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387edd5f-c28a-4732-ad0a-6af167143726-plugins-config-volume" pod "387edd5f-c28a-4732-ad0a-6af167143726" (UID: "387edd5f-c28a-4732-ad0a-6af167143726"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:10.085469 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.085394    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48daad3-b6ea-4e65-87dc-03a1c811a2cb-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:10.085469 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.085429    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brmrk\" (UniqueName: \"kubernetes.io/projected/a48daad3-b6ea-4e65-87dc-03a1c811a2cb-kube-api-access-brmrk\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:10.087286 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.087267    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387edd5f-c28a-4732-ad0a-6af167143726-kube-api-access-2cj8c" pod "387edd5f-c28a-4732-ad0a-6af167143726" (UID: "387edd5f-c28a-4732-ad0a-6af167143726"). InnerVolumeSpecName "kube-api-access-2cj8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:10.186821 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.186763    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387edd5f-c28a-4732-ad0a-6af167143726-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:10.186821 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.186788    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cj8c\" (UniqueName: \"kubernetes.io/projected/387edd5f-c28a-4732-ad0a-6af167143726-kube-api-access-2cj8c\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:10.266595 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.266566    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef41158-9775-4f62-a109-49e3d21649c9" path="/var/lib/kubelet/pods/3ef41158-9775-4f62-a109-49e3d21649c9/volumes"
Oct 30 19:21:10.394749 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.394723    4372 generic.go:356] "Generic (PLEG): container finished" podID="387edd5f-c28a-4732-ad0a-6af167143726" containerID="2ef2bc82034a2e592b05bd1060d996dfdbba9e823d6c0d58f38bd5310e2ea959" exitCode=0
Oct 30 19:21:10.394879 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.394767    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"
Oct 30 19:21:10.394879 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.394808    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm" event={"ID":"387edd5f-c28a-4732-ad0a-6af167143726","Type":"ContainerDied","Data":"2ef2bc82034a2e592b05bd1060d996dfdbba9e823d6c0d58f38bd5310e2ea959"}
Oct 30 19:21:10.394879 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.394863    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm" event={"ID":"387edd5f-c28a-4732-ad0a-6af167143726","Type":"ContainerDied","Data":"39887f82698247bb7d399e0590c731e2b22ff20a686a094d11b0c45c84f14b5b"}
Oct 30 19:21:10.395055 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.394892    4372 scope.go:122] "RemoveContainer" containerID="2ef2bc82034a2e592b05bd1060d996dfdbba9e823d6c0d58f38bd5310e2ea959"
Oct 30 19:21:10.396336 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.396314    4372 generic.go:356] "Generic (PLEG): container finished" podID="a48daad3-b6ea-4e65-87dc-03a1c811a2cb" containerID="4e800aee892fbcca4a611a51e12f827471d6dba87e178de1629bfda73682c6b8" exitCode=0
Oct 30 19:21:10.396431 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.396364    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"
Oct 30 19:21:10.396431 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.396388    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn" event={"ID":"a48daad3-b6ea-4e65-87dc-03a1c811a2cb","Type":"ContainerDied","Data":"4e800aee892fbcca4a611a51e12f827471d6dba87e178de1629bfda73682c6b8"}
Oct 30 19:21:10.396545 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.396438    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn" event={"ID":"a48daad3-b6ea-4e65-87dc-03a1c811a2cb","Type":"ContainerDied","Data":"15ab5b257f673ae5ea69384a28a4698f25f98ad29c64e9e45010e6c01657b01c"}
Oct 30 19:21:10.398340 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.398314    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx" event={"ID":"4934163b-b3eb-4f34-acd6-39a847daa0a8","Type":"ContainerStarted","Data":"6dacbb411356104ef410d60fd14928404baacfbd86f7c6da9aea6f079f631e09"}
Oct 30 19:21:10.398440 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.398364    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:10.398440 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.398384    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx" event={"ID":"4934163b-b3eb-4f34-acd6-39a847daa0a8","Type":"ContainerStarted","Data":"862cf4d51179ff6fc813ffea8899fe3d86169f0c2cf5881737f9e94955ca4f02"}
Oct 30 19:21:10.398603 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.398462    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:10.405633 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.405617    4372 scope.go:122] "RemoveContainer" containerID="2ef2bc82034a2e592b05bd1060d996dfdbba9e823d6c0d58f38bd5310e2ea959"
Oct 30 19:21:10.405922 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:10.405902    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef2bc82034a2e592b05bd1060d996dfdbba9e823d6c0d58f38bd5310e2ea959\": container with ID starting with 2ef2bc82034a2e592b05bd1060d996dfdbba9e823d6c0d58f38bd5310e2ea959 not found: ID does not exist" containerID="2ef2bc82034a2e592b05bd1060d996dfdbba9e823d6c0d58f38bd5310e2ea959"
Oct 30 19:21:10.405984 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.405928    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef2bc82034a2e592b05bd1060d996dfdbba9e823d6c0d58f38bd5310e2ea959"} err="failed to get container status \"2ef2bc82034a2e592b05bd1060d996dfdbba9e823d6c0d58f38bd5310e2ea959\": rpc error: code = NotFound desc = could not find container \"2ef2bc82034a2e592b05bd1060d996dfdbba9e823d6c0d58f38bd5310e2ea959\": container with ID starting with 2ef2bc82034a2e592b05bd1060d996dfdbba9e823d6c0d58f38bd5310e2ea959 not found: ID does not exist"
Oct 30 19:21:10.405984 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.405953    4372 scope.go:122] "RemoveContainer" containerID="4e800aee892fbcca4a611a51e12f827471d6dba87e178de1629bfda73682c6b8"
Oct 30 19:21:10.417498 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.417479    4372 scope.go:122] "RemoveContainer" containerID="4e800aee892fbcca4a611a51e12f827471d6dba87e178de1629bfda73682c6b8"
Oct 30 19:21:10.417822 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:10.417796    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e800aee892fbcca4a611a51e12f827471d6dba87e178de1629bfda73682c6b8\": container with ID starting with 4e800aee892fbcca4a611a51e12f827471d6dba87e178de1629bfda73682c6b8 not found: ID does not exist" containerID="4e800aee892fbcca4a611a51e12f827471d6dba87e178de1629bfda73682c6b8"
Oct 30 19:21:10.417916 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.417830    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e800aee892fbcca4a611a51e12f827471d6dba87e178de1629bfda73682c6b8"} err="failed to get container status \"4e800aee892fbcca4a611a51e12f827471d6dba87e178de1629bfda73682c6b8\": rpc error: code = NotFound desc = could not find container \"4e800aee892fbcca4a611a51e12f827471d6dba87e178de1629bfda73682c6b8\": container with ID starting with 4e800aee892fbcca4a611a51e12f827471d6dba87e178de1629bfda73682c6b8 not found: ID does not exist"
Oct 30 19:21:10.418372 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.418356    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"]
Oct 30 19:21:10.421766 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.421745    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7c6027-7f688dd58c-5xzrm"]
Oct 30 19:21:10.433332 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.433308    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"]
Oct 30 19:21:10.437905 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.437859    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7c6027-688d746fb4-srnxn"]
Oct 30 19:21:10.456151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.456082    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx" podStartSLOduration=1.456071065 podStartE2EDuration="1.456071065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:10.45435337 +0000 UTC m=+2884.322771713" watchObservedRunningTime="2026-10-30 19:21:10.456071065 +0000 UTC m=+2884.324489418"
Oct 30 19:21:10.480158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:10.480102    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl" podStartSLOduration=2.48009155 podStartE2EDuration="2.48009155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:10.477372826 +0000 UTC m=+2884.345791156" watchObservedRunningTime="2026-10-30 19:21:10.48009155 +0000 UTC m=+2884.348509907"
Oct 30 19:21:12.269754 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:12.269717    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="387edd5f-c28a-4732-ad0a-6af167143726" path="/var/lib/kubelet/pods/387edd5f-c28a-4732-ad0a-6af167143726/volumes"
Oct 30 19:21:12.270407 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:12.270384    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48daad3-b6ea-4e65-87dc-03a1c811a2cb" path="/var/lib/kubelet/pods/a48daad3-b6ea-4e65-87dc-03a1c811a2cb/volumes"
Oct 30 19:21:15.296037 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:15.295999    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"]
Oct 30 19:21:15.299509 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:15.299483    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:15.933982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:15.933947    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"]
Oct 30 19:21:15.936727 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:15.936701    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:19.504223 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:19.504193    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"]
Oct 30 19:21:19.504729 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:19.504373    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs" podUID="bd812973-9a0c-43a0-8bf2-982c4ef59bd2" containerName="maas-api" containerID="cri-o://5dfcda1a916ea385fb5edae213a6860a1a2cecf2db9a9d072f1064a309b5a3de" gracePeriod=30
Oct 30 19:21:19.643040 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:19.643002    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs" podUID="bd812973-9a0c-43a0-8bf2-982c4ef59bd2" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": dial tcp 10.133.0.168:8443: connect: connection refused"
Oct 30 19:21:19.943097 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:19.943067    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 19:21:20.000303 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.000270    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-openshift-service-ca\") pod \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") "
Oct 30 19:21:20.000694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.000654    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-openshift-service-ca" pod "bd812973-9a0c-43a0-8bf2-982c4ef59bd2" (UID: "bd812973-9a0c-43a0-8bf2-982c4ef59bd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:20.001040 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.001014    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-kube-api-access-dhdsx\" (UniqueName: \"kubernetes.io/projected/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-kube-api-access-dhdsx\") pod \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") "
Oct 30 19:21:20.001178 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.001115    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-metrics-certs\") pod \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") "
Oct 30 19:21:20.001295 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.001263    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-maas-api-tls\") pod \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\" (UID: \"bd812973-9a0c-43a0-8bf2-982c4ef59bd2\") "
Oct 30 19:21:20.001897 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.001870    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:20.004267 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.004234    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-metrics-certs" pod "bd812973-9a0c-43a0-8bf2-982c4ef59bd2" (UID: "bd812973-9a0c-43a0-8bf2-982c4ef59bd2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:20.004461 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.004432    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-maas-api-tls" pod "bd812973-9a0c-43a0-8bf2-982c4ef59bd2" (UID: "bd812973-9a0c-43a0-8bf2-982c4ef59bd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:20.005048 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.005021    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-kube-api-access-dhdsx" pod "bd812973-9a0c-43a0-8bf2-982c4ef59bd2" (UID: "bd812973-9a0c-43a0-8bf2-982c4ef59bd2"). InnerVolumeSpecName "kube-api-access-dhdsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:20.111733 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.109268    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:20.111733 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.109307    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:20.111733 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.109323    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhdsx\" (UniqueName: \"kubernetes.io/projected/bd812973-9a0c-43a0-8bf2-982c4ef59bd2-kube-api-access-dhdsx\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:20.296960 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.296920    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl" podUID="869ccb82-8beb-464c-9032-9540154538f8" containerName="payload-processing" containerID="cri-o://065c94441bf8234e9527459961f944f8419a7d5e180abd0f0f35514fdd1219d3" gracePeriod=25
Oct 30 19:21:20.503259 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.502957    4372 generic.go:356] "Generic (PLEG): container finished" podID="bd812973-9a0c-43a0-8bf2-982c4ef59bd2" containerID="5dfcda1a916ea385fb5edae213a6860a1a2cecf2db9a9d072f1064a309b5a3de" exitCode=0
Oct 30 19:21:20.503259 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.503025    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"
Oct 30 19:21:20.503488 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.503435    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs" event={"ID":"bd812973-9a0c-43a0-8bf2-982c4ef59bd2","Type":"ContainerDied","Data":"5dfcda1a916ea385fb5edae213a6860a1a2cecf2db9a9d072f1064a309b5a3de"}
Oct 30 19:21:20.503546 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.503490    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs" event={"ID":"bd812973-9a0c-43a0-8bf2-982c4ef59bd2","Type":"ContainerDied","Data":"def3dfb65017bb7eed89dd4ae733f0aca894662e9b908d62f9060eac6f890ac9"}
Oct 30 19:21:20.503546 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.503523    4372 scope.go:122] "RemoveContainer" containerID="5dfcda1a916ea385fb5edae213a6860a1a2cecf2db9a9d072f1064a309b5a3de"
Oct 30 19:21:20.507004 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.506963    4372 generic.go:356] "Generic (PLEG): container finished" podID="869ccb82-8beb-464c-9032-9540154538f8" containerID="065c94441bf8234e9527459961f944f8419a7d5e180abd0f0f35514fdd1219d3" exitCode=0
Oct 30 19:21:20.507519 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.507035    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl" event={"ID":"869ccb82-8beb-464c-9032-9540154538f8","Type":"ContainerDied","Data":"065c94441bf8234e9527459961f944f8419a7d5e180abd0f0f35514fdd1219d3"}
Oct 30 19:21:20.516714 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.516687    4372 scope.go:122] "RemoveContainer" containerID="5dfcda1a916ea385fb5edae213a6860a1a2cecf2db9a9d072f1064a309b5a3de"
Oct 30 19:21:20.517112 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:20.517089    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dfcda1a916ea385fb5edae213a6860a1a2cecf2db9a9d072f1064a309b5a3de\": container with ID starting with 5dfcda1a916ea385fb5edae213a6860a1a2cecf2db9a9d072f1064a309b5a3de not found: ID does not exist" containerID="5dfcda1a916ea385fb5edae213a6860a1a2cecf2db9a9d072f1064a309b5a3de"
Oct 30 19:21:20.517222 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.517186    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dfcda1a916ea385fb5edae213a6860a1a2cecf2db9a9d072f1064a309b5a3de"} err="failed to get container status \"5dfcda1a916ea385fb5edae213a6860a1a2cecf2db9a9d072f1064a309b5a3de\": rpc error: code = NotFound desc = could not find container \"5dfcda1a916ea385fb5edae213a6860a1a2cecf2db9a9d072f1064a309b5a3de\": container with ID starting with 5dfcda1a916ea385fb5edae213a6860a1a2cecf2db9a9d072f1064a309b5a3de not found: ID does not exist"
Oct 30 19:21:20.533752 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.533730    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"]
Oct 30 19:21:20.538776 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.538755    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-304f22-5497946bc4-fvgfs"]
Oct 30 19:21:20.574656 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.574640    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:20.717538 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.717510    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/869ccb82-8beb-464c-9032-9540154538f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/869ccb82-8beb-464c-9032-9540154538f8-plugins-config-volume\") pod \"869ccb82-8beb-464c-9032-9540154538f8\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") "
Oct 30 19:21:20.717693 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.717656    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/869ccb82-8beb-464c-9032-9540154538f8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/869ccb82-8beb-464c-9032-9540154538f8-routing-overlay-volume\") pod \"869ccb82-8beb-464c-9032-9540154538f8\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") "
Oct 30 19:21:20.717693 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.717684    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869ccb82-8beb-464c-9032-9540154538f8-kube-api-access-65vn2\" (UniqueName: \"kubernetes.io/projected/869ccb82-8beb-464c-9032-9540154538f8-kube-api-access-65vn2\") pod \"869ccb82-8beb-464c-9032-9540154538f8\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") "
Oct 30 19:21:20.717843 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.717823    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869ccb82-8beb-464c-9032-9540154538f8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/869ccb82-8beb-464c-9032-9540154538f8-provider-credentials\") pod \"869ccb82-8beb-464c-9032-9540154538f8\" (UID: \"869ccb82-8beb-464c-9032-9540154538f8\") "
Oct 30 19:21:20.718011 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.717987    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869ccb82-8beb-464c-9032-9540154538f8-plugins-config-volume" pod "869ccb82-8beb-464c-9032-9540154538f8" (UID: "869ccb82-8beb-464c-9032-9540154538f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:20.718078 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.718057    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869ccb82-8beb-464c-9032-9540154538f8-routing-overlay-volume" pod "869ccb82-8beb-464c-9032-9540154538f8" (UID: "869ccb82-8beb-464c-9032-9540154538f8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:20.718421 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.718400    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/869ccb82-8beb-464c-9032-9540154538f8-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:20.718482 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.718424    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/869ccb82-8beb-464c-9032-9540154538f8-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:20.720119 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.720096    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869ccb82-8beb-464c-9032-9540154538f8-kube-api-access-65vn2" pod "869ccb82-8beb-464c-9032-9540154538f8" (UID: "869ccb82-8beb-464c-9032-9540154538f8"). InnerVolumeSpecName "kube-api-access-65vn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:20.720197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.720165    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869ccb82-8beb-464c-9032-9540154538f8-provider-credentials" pod "869ccb82-8beb-464c-9032-9540154538f8" (UID: "869ccb82-8beb-464c-9032-9540154538f8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:20.819514 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.819439    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65vn2\" (UniqueName: \"kubernetes.io/projected/869ccb82-8beb-464c-9032-9540154538f8-kube-api-access-65vn2\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:20.819514 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.819467    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/869ccb82-8beb-464c-9032-9540154538f8-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:20.934937 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:20.934887    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx" podUID="4934163b-b3eb-4f34-acd6-39a847daa0a8" containerName="payload-processing" containerID="cri-o://6dacbb411356104ef410d60fd14928404baacfbd86f7c6da9aea6f079f631e09" gracePeriod=25
Oct 30 19:21:21.185116 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.185091    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:21.327554 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.327527    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4934163b-b3eb-4f34-acd6-39a847daa0a8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4934163b-b3eb-4f34-acd6-39a847daa0a8-routing-overlay-volume\") pod \"4934163b-b3eb-4f34-acd6-39a847daa0a8\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") "
Oct 30 19:21:21.327712 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.327623    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4934163b-b3eb-4f34-acd6-39a847daa0a8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4934163b-b3eb-4f34-acd6-39a847daa0a8-provider-credentials\") pod \"4934163b-b3eb-4f34-acd6-39a847daa0a8\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") "
Oct 30 19:21:21.327712 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.327659    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4934163b-b3eb-4f34-acd6-39a847daa0a8-kube-api-access-t48vw\" (UniqueName: \"kubernetes.io/projected/4934163b-b3eb-4f34-acd6-39a847daa0a8-kube-api-access-t48vw\") pod \"4934163b-b3eb-4f34-acd6-39a847daa0a8\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") "
Oct 30 19:21:21.327830 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.327740    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4934163b-b3eb-4f34-acd6-39a847daa0a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4934163b-b3eb-4f34-acd6-39a847daa0a8-plugins-config-volume\") pod \"4934163b-b3eb-4f34-acd6-39a847daa0a8\" (UID: \"4934163b-b3eb-4f34-acd6-39a847daa0a8\") "
Oct 30 19:21:21.327974 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.327950    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4934163b-b3eb-4f34-acd6-39a847daa0a8-routing-overlay-volume" pod "4934163b-b3eb-4f34-acd6-39a847daa0a8" (UID: "4934163b-b3eb-4f34-acd6-39a847daa0a8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:21.328276 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.328237    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4934163b-b3eb-4f34-acd6-39a847daa0a8-plugins-config-volume" pod "4934163b-b3eb-4f34-acd6-39a847daa0a8" (UID: "4934163b-b3eb-4f34-acd6-39a847daa0a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:21.328463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.328443    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4934163b-b3eb-4f34-acd6-39a847daa0a8-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:21.328519 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.328469    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4934163b-b3eb-4f34-acd6-39a847daa0a8-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:21.330609 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.330587    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4934163b-b3eb-4f34-acd6-39a847daa0a8-kube-api-access-t48vw" pod "4934163b-b3eb-4f34-acd6-39a847daa0a8" (UID: "4934163b-b3eb-4f34-acd6-39a847daa0a8"). InnerVolumeSpecName "kube-api-access-t48vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:21.330677 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.330601    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4934163b-b3eb-4f34-acd6-39a847daa0a8-provider-credentials" pod "4934163b-b3eb-4f34-acd6-39a847daa0a8" (UID: "4934163b-b3eb-4f34-acd6-39a847daa0a8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:21.429744 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.429662    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4934163b-b3eb-4f34-acd6-39a847daa0a8-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:21.429744 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.429693    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t48vw\" (UniqueName: \"kubernetes.io/projected/4934163b-b3eb-4f34-acd6-39a847daa0a8-kube-api-access-t48vw\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:21.516174 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.516142    4372 generic.go:356] "Generic (PLEG): container finished" podID="4934163b-b3eb-4f34-acd6-39a847daa0a8" containerID="6dacbb411356104ef410d60fd14928404baacfbd86f7c6da9aea6f079f631e09" exitCode=0
Oct 30 19:21:21.516640 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.516225    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"
Oct 30 19:21:21.516640 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.516245    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx" event={"ID":"4934163b-b3eb-4f34-acd6-39a847daa0a8","Type":"ContainerDied","Data":"6dacbb411356104ef410d60fd14928404baacfbd86f7c6da9aea6f079f631e09"}
Oct 30 19:21:21.516640 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.516307    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx" event={"ID":"4934163b-b3eb-4f34-acd6-39a847daa0a8","Type":"ContainerDied","Data":"862cf4d51179ff6fc813ffea8899fe3d86169f0c2cf5881737f9e94955ca4f02"}
Oct 30 19:21:21.516640 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.516339    4372 scope.go:122] "RemoveContainer" containerID="6dacbb411356104ef410d60fd14928404baacfbd86f7c6da9aea6f079f631e09"
Oct 30 19:21:21.523826 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.523805    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"
Oct 30 19:21:21.523826 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.523814    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl" event={"ID":"869ccb82-8beb-464c-9032-9540154538f8","Type":"ContainerDied","Data":"835aea4dbf712f7e2b0ce2e9e6bf2ef5a977f4d37dacc4992c827fdbaee55767"}
Oct 30 19:21:21.540197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.540176    4372 scope.go:122] "RemoveContainer" containerID="6dacbb411356104ef410d60fd14928404baacfbd86f7c6da9aea6f079f631e09"
Oct 30 19:21:21.540477 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:21.540456    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dacbb411356104ef410d60fd14928404baacfbd86f7c6da9aea6f079f631e09\": container with ID starting with 6dacbb411356104ef410d60fd14928404baacfbd86f7c6da9aea6f079f631e09 not found: ID does not exist" containerID="6dacbb411356104ef410d60fd14928404baacfbd86f7c6da9aea6f079f631e09"
Oct 30 19:21:21.540522 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.540486    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dacbb411356104ef410d60fd14928404baacfbd86f7c6da9aea6f079f631e09"} err="failed to get container status \"6dacbb411356104ef410d60fd14928404baacfbd86f7c6da9aea6f079f631e09\": rpc error: code = NotFound desc = could not find container \"6dacbb411356104ef410d60fd14928404baacfbd86f7c6da9aea6f079f631e09\": container with ID starting with 6dacbb411356104ef410d60fd14928404baacfbd86f7c6da9aea6f079f631e09 not found: ID does not exist"
Oct 30 19:21:21.540522 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.540501    4372 scope.go:122] "RemoveContainer" containerID="065c94441bf8234e9527459961f944f8419a7d5e180abd0f0f35514fdd1219d3"
Oct 30 19:21:21.558833 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.558810    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"]
Oct 30 19:21:21.565814 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.565789    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-84f9jw2fx"]
Oct 30 19:21:21.581137 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.581103    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"]
Oct 30 19:21:21.587079 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:21.587058    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-6bfcm9hpl"]
Oct 30 19:21:22.266677 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:22.266641    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4934163b-b3eb-4f34-acd6-39a847daa0a8" path="/var/lib/kubelet/pods/4934163b-b3eb-4f34-acd6-39a847daa0a8/volumes"
Oct 30 19:21:22.267154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:22.267114    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="869ccb82-8beb-464c-9032-9540154538f8" path="/var/lib/kubelet/pods/869ccb82-8beb-464c-9032-9540154538f8/volumes"
Oct 30 19:21:22.267795 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:22.267777    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd812973-9a0c-43a0-8bf2-982c4ef59bd2" path="/var/lib/kubelet/pods/bd812973-9a0c-43a0-8bf2-982c4ef59bd2/volumes"
Oct 30 19:21:22.410791 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:22.410759    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"]
Oct 30 19:21:22.410999 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:22.410976    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6" podUID="eb01e2fe-8077-4f8b-98e2-b107af18e2d4" containerName="istio-proxy" containerID="cri-o://9fa075200e13e549bb073ba5cc6b16e5cb8d08cd4fb71ae02d9620f433fbf500" gracePeriod=30
Oct 30 19:21:22.668181 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:22.668062    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main" containerID="cri-o://49052149934e225e1ddba240f28f8f7183373c5b0525de129c01309d9d384087" gracePeriod=45
Oct 30 19:21:22.943939 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:22.943911    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 19:21:23.067508 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.067468    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-home\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-home\") pod \"b2b4e963-9553-43b4-81da-5787cdfd3753\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") "
Oct 30 19:21:23.067703 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.067651    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-model-cache\") pod \"b2b4e963-9553-43b4-81da-5787cdfd3753\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") "
Oct 30 19:21:23.067790 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.067722    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-dshm\") pod \"b2b4e963-9553-43b4-81da-5787cdfd3753\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") "
Oct 30 19:21:23.067790 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.067764    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-home" pod "b2b4e963-9553-43b4-81da-5787cdfd3753" (UID: "b2b4e963-9553-43b4-81da-5787cdfd3753"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:23.067904 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.067776    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-tmp-dir\") pod \"b2b4e963-9553-43b4-81da-5787cdfd3753\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") "
Oct 30 19:21:23.067904 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.067866    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2b4e963-9553-43b4-81da-5787cdfd3753-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b4e963-9553-43b4-81da-5787cdfd3753-tls-certs\") pod \"b2b4e963-9553-43b4-81da-5787cdfd3753\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") "
Oct 30 19:21:23.068020 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.067920    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b4e963-9553-43b4-81da-5787cdfd3753-kube-api-access-dglx6\" (UniqueName: \"kubernetes.io/projected/b2b4e963-9553-43b4-81da-5787cdfd3753-kube-api-access-dglx6\") pod \"b2b4e963-9553-43b4-81da-5787cdfd3753\" (UID: \"b2b4e963-9553-43b4-81da-5787cdfd3753\") "
Oct 30 19:21:23.068020 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.068004    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-tmp-dir" pod "b2b4e963-9553-43b4-81da-5787cdfd3753" (UID: "b2b4e963-9553-43b4-81da-5787cdfd3753"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:23.068674 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.068215    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-model-cache" pod "b2b4e963-9553-43b4-81da-5787cdfd3753" (UID: "b2b4e963-9553-43b4-81da-5787cdfd3753"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:23.068674 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.068586    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:23.068674 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.068635    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:23.068674 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.068650    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:23.071513 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.071486    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b4e963-9553-43b4-81da-5787cdfd3753-kube-api-access-dglx6" pod "b2b4e963-9553-43b4-81da-5787cdfd3753" (UID: "b2b4e963-9553-43b4-81da-5787cdfd3753"). InnerVolumeSpecName "kube-api-access-dglx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:23.079556 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.079521    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-dshm" pod "b2b4e963-9553-43b4-81da-5787cdfd3753" (UID: "b2b4e963-9553-43b4-81da-5787cdfd3753"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:23.096253 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.096220    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b4e963-9553-43b4-81da-5787cdfd3753-tls-certs" pod "b2b4e963-9553-43b4-81da-5787cdfd3753" (UID: "b2b4e963-9553-43b4-81da-5787cdfd3753"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:23.170247 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.170221    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b4e963-9553-43b4-81da-5787cdfd3753-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:23.170247 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.170246    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b4e963-9553-43b4-81da-5787cdfd3753-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:23.170423 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.170256    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dglx6\" (UniqueName: \"kubernetes.io/projected/b2b4e963-9553-43b4-81da-5787cdfd3753-kube-api-access-dglx6\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:23.190415 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.190343    4372 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:21:23.190415 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.190401    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6" podUID="eb01e2fe-8077-4f8b-98e2-b107af18e2d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:21:23.577576 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.577536    4372 generic.go:356] "Generic (PLEG): container finished" podID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerID="49052149934e225e1ddba240f28f8f7183373c5b0525de129c01309d9d384087" exitCode=0
Oct 30 19:21:23.577762 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.577579    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"
Oct 30 19:21:23.577762 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.577618    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" event={"ID":"b2b4e963-9553-43b4-81da-5787cdfd3753","Type":"ContainerDied","Data":"49052149934e225e1ddba240f28f8f7183373c5b0525de129c01309d9d384087"}
Oct 30 19:21:23.577762 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.577670    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4" event={"ID":"b2b4e963-9553-43b4-81da-5787cdfd3753","Type":"ContainerDied","Data":"4dcdab83f5db75dd7ed9f38fa0d1121902ba1df7c0fd749e8a107d90736c3ab3"}
Oct 30 19:21:23.577762 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.577693    4372 scope.go:122] "RemoveContainer" containerID="49052149934e225e1ddba240f28f8f7183373c5b0525de129c01309d9d384087"
Oct 30 19:21:23.588510 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.588487    4372 scope.go:122] "RemoveContainer" containerID="49052149934e225e1ddba240f28f8f7183373c5b0525de129c01309d9d384087"
Oct 30 19:21:23.588777 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:23.588755    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49052149934e225e1ddba240f28f8f7183373c5b0525de129c01309d9d384087\": container with ID starting with 49052149934e225e1ddba240f28f8f7183373c5b0525de129c01309d9d384087 not found: ID does not exist" containerID="49052149934e225e1ddba240f28f8f7183373c5b0525de129c01309d9d384087"
Oct 30 19:21:23.588814 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.588786    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49052149934e225e1ddba240f28f8f7183373c5b0525de129c01309d9d384087"} err="failed to get container status \"49052149934e225e1ddba240f28f8f7183373c5b0525de129c01309d9d384087\": rpc error: code = NotFound desc = could not find container \"49052149934e225e1ddba240f28f8f7183373c5b0525de129c01309d9d384087\": container with ID starting with 49052149934e225e1ddba240f28f8f7183373c5b0525de129c01309d9d384087 not found: ID does not exist"
Oct 30 19:21:23.626712 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.626688    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"]
Oct 30 19:21:23.634068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:23.634042    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/distinct-2-w1-aa6736-kserve-fb85c8d4c-lg2h4"]
Oct 30 19:21:24.266034 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:24.266000    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" path="/var/lib/kubelet/pods/b2b4e963-9553-43b4-81da-5787cdfd3753/volumes"
Oct 30 19:21:27.611659 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.611627    4372 generic.go:356] "Generic (PLEG): container finished" podID="eb01e2fe-8077-4f8b-98e2-b107af18e2d4" containerID="9fa075200e13e549bb073ba5cc6b16e5cb8d08cd4fb71ae02d9620f433fbf500" exitCode=0
Oct 30 19:21:27.612045 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.611702    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6" event={"ID":"eb01e2fe-8077-4f8b-98e2-b107af18e2d4","Type":"ContainerDied","Data":"9fa075200e13e549bb073ba5cc6b16e5cb8d08cd4fb71ae02d9620f433fbf500"}
Oct 30 19:21:27.704266 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:27.704230    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:21:27.728215 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.728193    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:21:27.826504 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.826470    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-token\" (UniqueName: \"kubernetes.io/projected/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-token\") pod \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") "
Oct 30 19:21:27.826668 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.826605    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istiod-ca-cert\") pod \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") "
Oct 30 19:21:27.826767 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.826672    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-workload-socket\") pod \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") "
Oct 30 19:21:27.826767 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.826697    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-credential-socket\") pod \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") "
Oct 30 19:21:27.826893 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.826803    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-workload-certs\") pod \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") "
Oct 30 19:21:27.826893 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.826847    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-podinfo\") pod \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") "
Oct 30 19:21:27.826893 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.826873    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-kube-api-access-ck8hr\" (UniqueName: \"kubernetes.io/projected/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-kube-api-access-ck8hr\") pod \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") "
Oct 30 19:21:27.827051 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.826919    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-data\") pod \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") "
Oct 30 19:21:27.827051 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.826952    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-envoy\") pod \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\" (UID: \"eb01e2fe-8077-4f8b-98e2-b107af18e2d4\") "
Oct 30 19:21:27.827196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.827089    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-credential-socket" pod "eb01e2fe-8077-4f8b-98e2-b107af18e2d4" (UID: "eb01e2fe-8077-4f8b-98e2-b107af18e2d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:27.827196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.827099    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istiod-ca-cert" pod "eb01e2fe-8077-4f8b-98e2-b107af18e2d4" (UID: "eb01e2fe-8077-4f8b-98e2-b107af18e2d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:27.827196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.827104    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-workload-certs" pod "eb01e2fe-8077-4f8b-98e2-b107af18e2d4" (UID: "eb01e2fe-8077-4f8b-98e2-b107af18e2d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:27.827362 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.827315    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-data" pod "eb01e2fe-8077-4f8b-98e2-b107af18e2d4" (UID: "eb01e2fe-8077-4f8b-98e2-b107af18e2d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:27.827419 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.827382    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-workload-socket" pod "eb01e2fe-8077-4f8b-98e2-b107af18e2d4" (UID: "eb01e2fe-8077-4f8b-98e2-b107af18e2d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:27.827614 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.827590    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:27.827910 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.827619    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:27.827910 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.827636    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:27.827910 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.827651    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:27.827910 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.827664    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:27.828908 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.828884    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-token" pod "eb01e2fe-8077-4f8b-98e2-b107af18e2d4" (UID: "eb01e2fe-8077-4f8b-98e2-b107af18e2d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:27.829457 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.829429    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-podinfo" pod "eb01e2fe-8077-4f8b-98e2-b107af18e2d4" (UID: "eb01e2fe-8077-4f8b-98e2-b107af18e2d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:21:27.829457 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.829434    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-kube-api-access-ck8hr" pod "eb01e2fe-8077-4f8b-98e2-b107af18e2d4" (UID: "eb01e2fe-8077-4f8b-98e2-b107af18e2d4"). InnerVolumeSpecName "kube-api-access-ck8hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:27.829600 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.829520    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-envoy" pod "eb01e2fe-8077-4f8b-98e2-b107af18e2d4" (UID: "eb01e2fe-8077-4f8b-98e2-b107af18e2d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:27.857866 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.857844    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"]
Oct 30 19:21:27.928901 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.928876    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:27.928901 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.928900    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck8hr\" (UniqueName: \"kubernetes.io/projected/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-kube-api-access-ck8hr\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:27.929048 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.928911    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:27.929048 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:27.928921    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb01e2fe-8077-4f8b-98e2-b107af18e2d4-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:28.620430 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:28.620397    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"
Oct 30 19:21:28.620949 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:28.620403    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6" event={"ID":"eb01e2fe-8077-4f8b-98e2-b107af18e2d4","Type":"ContainerDied","Data":"f5ce1651f942913f03523aa78b2e9905707cf02cf1eeafbe1f983b3a57913b86"}
Oct 30 19:21:28.620949 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:28.620557    4372 scope.go:122] "RemoveContainer" containerID="9fa075200e13e549bb073ba5cc6b16e5cb8d08cd4fb71ae02d9620f433fbf500"
Oct 30 19:21:28.644009 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:28.643986    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"]
Oct 30 19:21:28.651451 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:28.651428    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03362d-openshift-default-68db56f9df-lvnz6"]
Oct 30 19:21:29.414658 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:29.414624    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"]
Oct 30 19:21:29.414867 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:29.414844    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv" podUID="13a85f8c-9244-499b-a12f-37d11fb59cba" containerName="istio-proxy" containerID="cri-o://297cc4497c14c70280ad7a9a78c604284c1811b68a13cdd5f93b4fefbcfabc08" gracePeriod=30
Oct 30 19:21:30.266341 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:30.266308    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb01e2fe-8077-4f8b-98e2-b107af18e2d4" path="/var/lib/kubelet/pods/eb01e2fe-8077-4f8b-98e2-b107af18e2d4/volumes"
Oct 30 19:21:34.674174 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.674105    4372 generic.go:356] "Generic (PLEG): container finished" podID="13a85f8c-9244-499b-a12f-37d11fb59cba" containerID="297cc4497c14c70280ad7a9a78c604284c1811b68a13cdd5f93b4fefbcfabc08" exitCode=0
Oct 30 19:21:34.675000 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.674214    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv" event={"ID":"13a85f8c-9244-499b-a12f-37d11fb59cba","Type":"ContainerDied","Data":"297cc4497c14c70280ad7a9a78c604284c1811b68a13cdd5f93b4fefbcfabc08"}
Oct 30 19:21:34.675000 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.674314    4372 scope.go:122] "RemoveContainer" containerID="048efa7ee353ef701682ffe878dd6679f7618964afceb5ae301a663401b692f4"
Oct 30 19:21:34.718636 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.718613    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 19:21:34.834509 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.834478    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13a85f8c-9244-499b-a12f-37d11fb59cba-kube-api-access-2t4n8\" (UniqueName: \"kubernetes.io/projected/13a85f8c-9244-499b-a12f-37d11fb59cba-kube-api-access-2t4n8\") pod \"13a85f8c-9244-499b-a12f-37d11fb59cba\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") "
Oct 30 19:21:34.834694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.834546    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-credential-socket\") pod \"13a85f8c-9244-499b-a12f-37d11fb59cba\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") "
Oct 30 19:21:34.834694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.834597    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-token\" (UniqueName: \"kubernetes.io/projected/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-token\") pod \"13a85f8c-9244-499b-a12f-37d11fb59cba\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") "
Oct 30 19:21:34.834694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.834632    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-data\") pod \"13a85f8c-9244-499b-a12f-37d11fb59cba\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") "
Oct 30 19:21:34.834909 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.834714    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-workload-socket\") pod \"13a85f8c-9244-499b-a12f-37d11fb59cba\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") "
Oct 30 19:21:34.834909 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.834753    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-podinfo\") pod \"13a85f8c-9244-499b-a12f-37d11fb59cba\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") "
Oct 30 19:21:34.834909 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.834785    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-workload-certs\") pod \"13a85f8c-9244-499b-a12f-37d11fb59cba\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") "
Oct 30 19:21:34.834909 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.834837    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-envoy\") pod \"13a85f8c-9244-499b-a12f-37d11fb59cba\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") "
Oct 30 19:21:34.834909 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.834871    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13a85f8c-9244-499b-a12f-37d11fb59cba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13a85f8c-9244-499b-a12f-37d11fb59cba-istiod-ca-cert\") pod \"13a85f8c-9244-499b-a12f-37d11fb59cba\" (UID: \"13a85f8c-9244-499b-a12f-37d11fb59cba\") "
Oct 30 19:21:34.835230 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.834952    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-credential-socket" pod "13a85f8c-9244-499b-a12f-37d11fb59cba" (UID: "13a85f8c-9244-499b-a12f-37d11fb59cba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:34.835230 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.835000    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-data" pod "13a85f8c-9244-499b-a12f-37d11fb59cba" (UID: "13a85f8c-9244-499b-a12f-37d11fb59cba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:34.835230 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.835200    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-workload-certs" pod "13a85f8c-9244-499b-a12f-37d11fb59cba" (UID: "13a85f8c-9244-499b-a12f-37d11fb59cba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:34.835422 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.835321    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-workload-socket" pod "13a85f8c-9244-499b-a12f-37d11fb59cba" (UID: "13a85f8c-9244-499b-a12f-37d11fb59cba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:34.835486 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.835428    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a85f8c-9244-499b-a12f-37d11fb59cba-istiod-ca-cert" pod "13a85f8c-9244-499b-a12f-37d11fb59cba" (UID: "13a85f8c-9244-499b-a12f-37d11fb59cba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:34.835775 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.835725    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:34.835878 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.835781    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:34.835878 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.835795    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:34.835878 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.835808    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13a85f8c-9244-499b-a12f-37d11fb59cba-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:34.835878 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.835823    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:34.837182 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.837096    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a85f8c-9244-499b-a12f-37d11fb59cba-kube-api-access-2t4n8" pod "13a85f8c-9244-499b-a12f-37d11fb59cba" (UID: "13a85f8c-9244-499b-a12f-37d11fb59cba"). InnerVolumeSpecName "kube-api-access-2t4n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:34.837651 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.837628    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-podinfo" pod "13a85f8c-9244-499b-a12f-37d11fb59cba" (UID: "13a85f8c-9244-499b-a12f-37d11fb59cba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:21:34.837819 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.837799    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-token" pod "13a85f8c-9244-499b-a12f-37d11fb59cba" (UID: "13a85f8c-9244-499b-a12f-37d11fb59cba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:34.838083 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.838062    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-envoy" pod "13a85f8c-9244-499b-a12f-37d11fb59cba" (UID: "13a85f8c-9244-499b-a12f-37d11fb59cba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:34.937234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.937196    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:34.937234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.937235    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:34.937390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.937249    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t4n8\" (UniqueName: \"kubernetes.io/projected/13a85f8c-9244-499b-a12f-37d11fb59cba-kube-api-access-2t4n8\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:34.937390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:34.937264    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13a85f8c-9244-499b-a12f-37d11fb59cba-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:35.262752 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:35.262723    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:21:35.684812 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:35.684782    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"
Oct 30 19:21:35.684812 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:35.684801    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv" event={"ID":"13a85f8c-9244-499b-a12f-37d11fb59cba","Type":"ContainerDied","Data":"7a333bc394ff692d19e7029195d1f2ae2b70b2eb2ddd599ea134ff43a30a36ea"}
Oct 30 19:21:35.685328 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:35.684855    4372 scope.go:122] "RemoveContainer" containerID="297cc4497c14c70280ad7a9a78c604284c1811b68a13cdd5f93b4fefbcfabc08"
Oct 30 19:21:35.723342 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:35.723309    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"]
Oct 30 19:21:35.750143 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:35.750099    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-304f22-openshift-default-5d464f7f5c-dw5pv"]
Oct 30 19:21:36.265406 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:36.265378    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13a85f8c-9244-499b-a12f-37d11fb59cba" path="/var/lib/kubelet/pods/13a85f8c-9244-499b-a12f-37d11fb59cba/volumes"
Oct 30 19:21:38.186625 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.186589    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"]
Oct 30 19:21:38.188169 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.188141    4372 cpu_manager.go:414] "containerMap: removing container" podUID="13a85f8c-9244-499b-a12f-37d11fb59cba" containerName="istio-proxy"
Oct 30 19:21:38.188351 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.188335    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a85f8c-9244-499b-a12f-37d11fb59cba" containerName="istio-proxy"
Oct 30 19:21:38.188474 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.188461    4372 cpu_manager.go:414] "containerMap: removing container" podUID="eb01e2fe-8077-4f8b-98e2-b107af18e2d4" containerName="istio-proxy"
Oct 30 19:21:38.188577 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.188564    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb01e2fe-8077-4f8b-98e2-b107af18e2d4" containerName="istio-proxy"
Oct 30 19:21:38.188710 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.188696    4372 cpu_manager.go:414] "containerMap: removing container" podUID="bd812973-9a0c-43a0-8bf2-982c4ef59bd2" containerName="maas-api"
Oct 30 19:21:38.188814 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.188801    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd812973-9a0c-43a0-8bf2-982c4ef59bd2" containerName="maas-api"
Oct 30 19:21:38.188902 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.188891    4372 cpu_manager.go:414] "containerMap: removing container" podUID="a48daad3-b6ea-4e65-87dc-03a1c811a2cb" containerName="payload-processing"
Oct 30 19:21:38.188992 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.188981    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48daad3-b6ea-4e65-87dc-03a1c811a2cb" containerName="payload-processing"
Oct 30 19:21:38.189101 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.189089    4372 cpu_manager.go:414] "containerMap: removing container" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main"
Oct 30 19:21:38.189206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.189193    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main"
Oct 30 19:21:38.189310 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.189294    4372 cpu_manager.go:414] "containerMap: removing container" podUID="387edd5f-c28a-4732-ad0a-6af167143726" containerName="payload-pre-processing"
Oct 30 19:21:38.189409 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.189395    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387edd5f-c28a-4732-ad0a-6af167143726" containerName="payload-pre-processing"
Oct 30 19:21:38.189503 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.189493    4372 cpu_manager.go:414] "containerMap: removing container" podUID="869ccb82-8beb-464c-9032-9540154538f8" containerName="payload-processing"
Oct 30 19:21:38.189590 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.189578    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="869ccb82-8beb-464c-9032-9540154538f8" containerName="payload-processing"
Oct 30 19:21:38.189699 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.189687    4372 cpu_manager.go:414] "containerMap: removing container" podUID="4934163b-b3eb-4f34-acd6-39a847daa0a8" containerName="payload-processing"
Oct 30 19:21:38.189795 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.189783    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4934163b-b3eb-4f34-acd6-39a847daa0a8" containerName="payload-processing"
Oct 30 19:21:38.190044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.190027    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd812973-9a0c-43a0-8bf2-982c4ef59bd2" containerName="maas-api"
Oct 30 19:21:38.190310 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.190292    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48daad3-b6ea-4e65-87dc-03a1c811a2cb" containerName="payload-processing"
Oct 30 19:21:38.190446 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.190431    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b4e963-9553-43b4-81da-5787cdfd3753" containerName="main"
Oct 30 19:21:38.190658 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.190624    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="387edd5f-c28a-4732-ad0a-6af167143726" containerName="payload-pre-processing"
Oct 30 19:21:38.190812 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.190781    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="869ccb82-8beb-464c-9032-9540154538f8" containerName="payload-processing"
Oct 30 19:21:38.190949 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.190936    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a85f8c-9244-499b-a12f-37d11fb59cba" containerName="istio-proxy"
Oct 30 19:21:38.191092 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.191078    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="4934163b-b3eb-4f34-acd6-39a847daa0a8" containerName="payload-processing"
Oct 30 19:21:38.191259 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.191233    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a85f8c-9244-499b-a12f-37d11fb59cba" containerName="istio-proxy"
Oct 30 19:21:38.191430 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.191415    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb01e2fe-8077-4f8b-98e2-b107af18e2d4" containerName="istio-proxy"
Oct 30 19:21:38.200592 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.200566    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:38.206212 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.206184    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"]
Oct 30 19:21:38.212023 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.212002    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"agc-cred-676f802b0d13\""
Oct 30 19:21:38.213080 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.213057    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"payload-processing-external-model-e2e--3f19eb37-dockercfg-phgbc\""
Oct 30 19:21:38.213235 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.213143    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"kube-root-ca.crt\""
Oct 30 19:21:38.213440 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.213416    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"payload-processing-external-model-plugins-e2e-extmod-a-21d896\""
Oct 30 19:21:38.213526 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.213293    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"routing-overlay\""
Oct 30 19:21:38.213633 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.213612    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"openshift-service-ca.crt\""
Oct 30 19:21:38.292944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.292918    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:38.293096 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.292960    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xd47\" (UniqueName: \"kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-kube-api-access-4xd47\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:38.293096 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.293039    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:38.293311 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.293285    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:38.394283 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.394253    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:38.394446 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.394338    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xd47\" (UniqueName: \"kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-kube-api-access-4xd47\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:38.394446 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.394390    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:38.394628 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.394604    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:38.394705 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:38.394681    4372 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d72beb/agc-cred-919331a7b490: secret "agc-cred-919331a7b490" not found
Oct 30 19:21:38.394705 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:38.394700    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl: secret "agc-cred-919331a7b490" not found
Oct 30 19:21:38.394835 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:38.394757    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-provider-credentials podName:88adeb15-521e-4bf1-85bd-6b549ea5c460 nodeName:}" failed. No retries permitted until 2026-10-30 19:21:38.894738632 +0000 UTC m=+2912.763156961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl" (UID: "88adeb15-521e-4bf1-85bd-6b549ea5c460") : secret "agc-cred-919331a7b490" not found
Oct 30 19:21:38.395205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.395180    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:38.395368 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.395350    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:38.397370 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.397348    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:38.410355 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.410332    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xd47\" (UniqueName: \"kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-kube-api-access-4xd47\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:38.524003 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.523938    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:38.696149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.684344    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"]
Oct 30 19:21:38.696149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.685229    4372 cpu_manager.go:414] "containerMap: removing container" podUID="13a85f8c-9244-499b-a12f-37d11fb59cba" containerName="istio-proxy"
Oct 30 19:21:38.696149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.685252    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a85f8c-9244-499b-a12f-37d11fb59cba" containerName="istio-proxy"
Oct 30 19:21:38.717501 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.717472    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:38.743912 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.743043    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"]
Oct 30 19:21:38.808447 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.808414    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"]
Oct 30 19:21:38.809407 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:21:38.809378    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86c2cb2_8ae4_4a42_b277_3779c70b995a.slice/crio-84ed9bae7e0815acc9f09fb09492da3aa2becc557ba32dac703b1801b2a3214a WatchSource:0}: Error finding container 84ed9bae7e0815acc9f09fb09492da3aa2becc557ba32dac703b1801b2a3214a: Status 404 returned error can't find the container with id 84ed9bae7e0815acc9f09fb09492da3aa2becc557ba32dac703b1801b2a3214a
Oct 30 19:21:38.850139 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.850099    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9nhn\" (UniqueName: \"kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-kube-api-access-r9nhn\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:38.850259 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.850239    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:38.850351 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.850331    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:38.850394 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.850368    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:38.952085 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.952060    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9nhn\" (UniqueName: \"kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-kube-api-access-r9nhn\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:38.952211 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.952161    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:38.952257 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.952214    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:38.952257 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.952239    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:38.952395 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:38.952379    4372 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d72beb/agc-cred-919331a7b490: secret "agc-cred-919331a7b490" not found
Oct 30 19:21:38.952441 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:38.952400    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl: secret "agc-cred-919331a7b490" not found
Oct 30 19:21:38.952477 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:38.952458    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-provider-credentials podName:88adeb15-521e-4bf1-85bd-6b549ea5c460 nodeName:}" failed. No retries permitted until 2026-10-30 19:21:39.952438345 +0000 UTC m=+2913.820856662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl" (UID: "88adeb15-521e-4bf1-85bd-6b549ea5c460") : secret "agc-cred-919331a7b490" not found
Oct 30 19:21:38.952939 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.952920    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:38.952982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.952947    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:38.954631 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.954611    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:38.965298 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:38.965279    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9nhn\" (UniqueName: \"kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-kube-api-access-r9nhn\") pod \"payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") " pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:39.074502 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:39.074480    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:39.359664 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:21:39.359636    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8447ca7_004b_480f_8d70_c1896a5aa6ca.slice/crio-923ec4f8e90c5a612007c0e7aa0ac424e4fc8d7dee09070245aafed4c882933f WatchSource:0}: Error finding container 923ec4f8e90c5a612007c0e7aa0ac424e4fc8d7dee09070245aafed4c882933f: Status 404 returned error can't find the container with id 923ec4f8e90c5a612007c0e7aa0ac424e4fc8d7dee09070245aafed4c882933f
Oct 30 19:21:39.360354 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:39.360327    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"]
Oct 30 19:21:39.517228 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:39.517187    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"]
Oct 30 19:21:39.806057 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:39.805959    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84" event={"ID":"b8447ca7-004b-480f-8d70-c1896a5aa6ca","Type":"ContainerStarted","Data":"f794317bdc56dfd2b32e16864ab0a6bdeeb28bd6685fa530fcd7ac9285a709a4"}
Oct 30 19:21:39.806353 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:39.806331    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84" event={"ID":"b8447ca7-004b-480f-8d70-c1896a5aa6ca","Type":"ContainerStarted","Data":"923ec4f8e90c5a612007c0e7aa0ac424e4fc8d7dee09070245aafed4c882933f"}
Oct 30 19:21:39.807500 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:39.807477    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:39.810244 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:39.810217    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86" event={"ID":"e86c2cb2-8ae4-4a42-b277-3779c70b995a","Type":"ContainerStarted","Data":"29572d3366e4f668d99664efd31e11dd1480a48027b60467e62db7f449cc571c"}
Oct 30 19:21:39.812576 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:39.812543    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86" event={"ID":"e86c2cb2-8ae4-4a42-b277-3779c70b995a","Type":"ContainerStarted","Data":"84ed9bae7e0815acc9f09fb09492da3aa2becc557ba32dac703b1801b2a3214a"}
Oct 30 19:21:39.812765 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:39.812749    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:39.812912 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:39.810698    4372 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86" secret="" err="secret \"payload-processing-external-model-e2e--3f19eb37-dockercfg-phgbc\" not found"
Oct 30 19:21:39.834170 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:39.833356    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84" podStartSLOduration=1.833343766 podStartE2EDuration="1.833343766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:39.830113888 +0000 UTC m=+2913.698532218" watchObservedRunningTime="2026-10-30 19:21:39.833343766 +0000 UTC m=+2913.701762096"
Oct 30 19:21:39.855729 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:39.855670    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86" podStartSLOduration=1.8556568549999999 podStartE2EDuration="1.855656855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:39.853379756 +0000 UTC m=+2913.721798112" watchObservedRunningTime="2026-10-30 19:21:39.855656855 +0000 UTC m=+2913.724075219"
Oct 30 19:21:39.903984 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:39.903946    4372 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-plugins-e2e-extmod-a-21d896: configmap "payload-processing-external-model-plugins-e2e-extmod-a-21d896" not found
Oct 30 19:21:39.904236 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:39.904035    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-plugins-config-volume podName:e86c2cb2-8ae4-4a42-b277-3779c70b995a nodeName:}" failed. No retries permitted until 2026-10-30 19:21:40.404013681 +0000 UTC m=+2914.272432008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86" (UID: "e86c2cb2-8ae4-4a42-b277-3779c70b995a") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-21d896" not found
Oct 30 19:21:39.904236 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:39.904097    4372 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d72beb/agc-cred-1a4160222042: secret "agc-cred-1a4160222042" not found
Oct 30 19:21:39.904236 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:39.904160    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84: secret "agc-cred-1a4160222042" not found
Oct 30 19:21:39.904236 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:39.904231    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-provider-credentials podName:b8447ca7-004b-480f-8d70-c1896a5aa6ca nodeName:}" failed. No retries permitted until 2026-10-30 19:21:40.404212148 +0000 UTC m=+2914.272630467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84" (UID: "b8447ca7-004b-480f-8d70-c1896a5aa6ca") : secret "agc-cred-1a4160222042" not found
Oct 30 19:21:39.904535 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:39.904242    4372 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-21d896/agc-cred-676f802b0d13: secret "agc-cred-676f802b0d13" not found
Oct 30 19:21:39.904535 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:39.904259    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86: secret "agc-cred-676f802b0d13" not found
Oct 30 19:21:39.904535 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:39.904299    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-provider-credentials podName:e86c2cb2-8ae4-4a42-b277-3779c70b995a nodeName:}" failed. No retries permitted until 2026-10-30 19:21:40.404286547 +0000 UTC m=+2914.272704876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86" (UID: "e86c2cb2-8ae4-4a42-b277-3779c70b995a") : secret "agc-cred-676f802b0d13" not found
Oct 30 19:21:39.994561 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:39.993937    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"]
Oct 30 19:21:39.997814 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:39.997786    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"]
Oct 30 19:21:40.005565 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:40.005539    4372 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d72beb/agc-cred-1a4160222042: secret "agc-cred-1a4160222042" not found
Oct 30 19:21:40.005565 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:40.005566    4372 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d72beb/agc-cred-919331a7b490: secret "agc-cred-919331a7b490" not found
Oct 30 19:21:40.005762 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:40.005579    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl: [secret "agc-cred-1a4160222042" not found, secret "agc-cred-919331a7b490" not found]
Oct 30 19:21:40.005762 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:40.005649    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-provider-credentials podName:88adeb15-521e-4bf1-85bd-6b549ea5c460 nodeName:}" failed. No retries permitted until 2026-10-30 19:21:42.005630695 +0000 UTC m=+2915.874049020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl" (UID: "88adeb15-521e-4bf1-85bd-6b549ea5c460") : [secret "agc-cred-1a4160222042" not found, secret "agc-cred-919331a7b490" not found]
Oct 30 19:21:40.290863 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.290831    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"]
Oct 30 19:21:40.305015 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.304988    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"]
Oct 30 19:21:40.375979 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.375933    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"]
Oct 30 19:21:40.376447 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.376138    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn" podUID="2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56" containerName="maas-api" containerID="cri-o://3fc087c0323837120364b8037fab21e88bad77aade2b9e95524b473844892fba" gracePeriod=30
Oct 30 19:21:40.430599 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:40.430566    4372 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-21d896/agc-cred-676f802b0d13: secret "agc-cred-676f802b0d13" not found
Oct 30 19:21:40.430734 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:40.430609    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86: secret "agc-cred-676f802b0d13" not found
Oct 30 19:21:40.430734 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:40.430649    4372 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d72beb/agc-cred-1a4160222042: secret "agc-cred-1a4160222042" not found
Oct 30 19:21:40.430734 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:40.430669    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84: secret "agc-cred-1a4160222042" not found
Oct 30 19:21:40.430734 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:40.430575    4372 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-plugins-e2e-extmod-a-21d896: configmap "payload-processing-external-model-plugins-e2e-extmod-a-21d896" not found
Oct 30 19:21:40.430734 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:40.430675    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-provider-credentials podName:e86c2cb2-8ae4-4a42-b277-3779c70b995a nodeName:}" failed. No retries permitted until 2026-10-30 19:21:41.430653604 +0000 UTC m=+2915.299071953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86" (UID: "e86c2cb2-8ae4-4a42-b277-3779c70b995a") : secret "agc-cred-676f802b0d13" not found
Oct 30 19:21:40.430947 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:40.430742    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-provider-credentials podName:b8447ca7-004b-480f-8d70-c1896a5aa6ca nodeName:}" failed. No retries permitted until 2026-10-30 19:21:41.430730283 +0000 UTC m=+2915.299148592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84" (UID: "b8447ca7-004b-480f-8d70-c1896a5aa6ca") : secret "agc-cred-1a4160222042" not found
Oct 30 19:21:40.430947 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:40.430754    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-plugins-config-volume podName:e86c2cb2-8ae4-4a42-b277-3779c70b995a nodeName:}" failed. No retries permitted until 2026-10-30 19:21:41.430748863 +0000 UTC m=+2915.299167172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86" (UID: "e86c2cb2-8ae4-4a42-b277-3779c70b995a") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-21d896" not found
Oct 30 19:21:40.769390 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.769360    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 19:21:40.819605 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.819517    4372 generic.go:356] "Generic (PLEG): container finished" podID="2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56" containerID="3fc087c0323837120364b8037fab21e88bad77aade2b9e95524b473844892fba" exitCode=0
Oct 30 19:21:40.819605 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.819568    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"
Oct 30 19:21:40.819820 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.819599    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn" event={"ID":"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56","Type":"ContainerDied","Data":"3fc087c0323837120364b8037fab21e88bad77aade2b9e95524b473844892fba"}
Oct 30 19:21:40.819820 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.819653    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn" event={"ID":"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56","Type":"ContainerDied","Data":"f911e090f974d96bd9a3e26dd834c6e216516e48903a5722e1e05cea15614a4a"}
Oct 30 19:21:40.819820 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.819683    4372 scope.go:122] "RemoveContainer" containerID="3fc087c0323837120364b8037fab21e88bad77aade2b9e95524b473844892fba"
Oct 30 19:21:40.820292 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.820265    4372 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84" secret="" err="secret \"payload-processing-external-model-e2e--b99c0f48-dockercfg-zjmbc\" not found"
Oct 30 19:21:40.831272 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.831249    4372 scope.go:122] "RemoveContainer" containerID="3fc087c0323837120364b8037fab21e88bad77aade2b9e95524b473844892fba"
Oct 30 19:21:40.831573 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:40.831550    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc087c0323837120364b8037fab21e88bad77aade2b9e95524b473844892fba\": container with ID starting with 3fc087c0323837120364b8037fab21e88bad77aade2b9e95524b473844892fba not found: ID does not exist" containerID="3fc087c0323837120364b8037fab21e88bad77aade2b9e95524b473844892fba"
Oct 30 19:21:40.831679 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.831581    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc087c0323837120364b8037fab21e88bad77aade2b9e95524b473844892fba"} err="failed to get container status \"3fc087c0323837120364b8037fab21e88bad77aade2b9e95524b473844892fba\": rpc error: code = NotFound desc = could not find container \"3fc087c0323837120364b8037fab21e88bad77aade2b9e95524b473844892fba\": container with ID starting with 3fc087c0323837120364b8037fab21e88bad77aade2b9e95524b473844892fba not found: ID does not exist"
Oct 30 19:21:40.867181 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.867139    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-kube-api-access-h8w4r\" (UniqueName: \"kubernetes.io/projected/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-kube-api-access-h8w4r\") pod \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") "
Oct 30 19:21:40.867316 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.867297    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-maas-api-tls\") pod \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") "
Oct 30 19:21:40.867481 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.867460    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-openshift-service-ca\") pod \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") "
Oct 30 19:21:40.867719 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.867627    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-metrics-certs\") pod \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\" (UID: \"2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56\") "
Oct 30 19:21:40.867799 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.867785    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-openshift-service-ca" pod "2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56" (UID: "2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:40.868060 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:40.868038    4372 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-plugins-e2e-extmod-b-d72beb: configmap "payload-processing-external-model-plugins-e2e-extmod-b-d72beb" not found
Oct 30 19:21:40.868215 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:40.868156    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-plugins-config-volume podName:b8447ca7-004b-480f-8d70-c1896a5aa6ca nodeName:}" failed. No retries permitted until 2026-10-30 19:21:41.368110309 +0000 UTC m=+2915.236528631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84" (UID: "b8447ca7-004b-480f-8d70-c1896a5aa6ca") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-d72beb" not found
Oct 30 19:21:40.868553 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.868519    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:40.869550 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.869522    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-kube-api-access-h8w4r" pod "2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56" (UID: "2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56"). InnerVolumeSpecName "kube-api-access-h8w4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:40.869713 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.869693    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-maas-api-tls" pod "2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56" (UID: "2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:40.869965 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.869943    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-metrics-certs" pod "2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56" (UID: "2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:40.969732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.969698    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:40.969732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.969732    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8w4r\" (UniqueName: \"kubernetes.io/projected/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-kube-api-access-h8w4r\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:40.969954 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:40.969746    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:41.146790 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:41.146761    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"]
Oct 30 19:21:41.149770 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:41.149740    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-aa6736-fb8bc975f-lp9fn"]
Oct 30 19:21:41.374382 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:41.374351    4372 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-plugins-e2e-extmod-b-d72beb: configmap "payload-processing-external-model-plugins-e2e-extmod-b-d72beb" not found
Oct 30 19:21:41.374544 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:41.374417    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-plugins-config-volume podName:b8447ca7-004b-480f-8d70-c1896a5aa6ca nodeName:}" failed. No retries permitted until 2026-10-30 19:21:42.374402072 +0000 UTC m=+2916.242820399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84" (UID: "b8447ca7-004b-480f-8d70-c1896a5aa6ca") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-d72beb" not found
Oct 30 19:21:41.475733 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:41.475660    4372 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-21d896/agc-cred-676f802b0d13: secret "agc-cred-676f802b0d13" not found
Oct 30 19:21:41.475733 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:41.475688    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86: secret "agc-cred-676f802b0d13" not found
Oct 30 19:21:41.475733 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:41.475689    4372 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-plugins-e2e-extmod-a-21d896: configmap "payload-processing-external-model-plugins-e2e-extmod-a-21d896" not found
Oct 30 19:21:41.475733 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:41.475722    4372 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d72beb/agc-cred-1a4160222042: secret "agc-cred-1a4160222042" not found
Oct 30 19:21:41.476242 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:41.475742    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84: secret "agc-cred-1a4160222042" not found
Oct 30 19:21:41.476242 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:41.475744    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-plugins-config-volume podName:e86c2cb2-8ae4-4a42-b277-3779c70b995a nodeName:}" failed. No retries permitted until 2026-10-30 19:21:43.475723564 +0000 UTC m=+2917.344141894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86" (UID: "e86c2cb2-8ae4-4a42-b277-3779c70b995a") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-21d896" not found
Oct 30 19:21:41.476242 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:41.475767    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-provider-credentials podName:e86c2cb2-8ae4-4a42-b277-3779c70b995a nodeName:}" failed. No retries permitted until 2026-10-30 19:21:43.475757406 +0000 UTC m=+2917.344175714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86" (UID: "e86c2cb2-8ae4-4a42-b277-3779c70b995a") : secret "agc-cred-676f802b0d13" not found
Oct 30 19:21:41.476242 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:41.475788    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-provider-credentials podName:b8447ca7-004b-480f-8d70-c1896a5aa6ca nodeName:}" failed. No retries permitted until 2026-10-30 19:21:43.475776785 +0000 UTC m=+2917.344195093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84" (UID: "b8447ca7-004b-480f-8d70-c1896a5aa6ca") : secret "agc-cred-1a4160222042" not found
Oct 30 19:21:42.085535 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:42.085507    4372 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d72beb/agc-cred-1a4160222042: secret "agc-cred-1a4160222042" not found
Oct 30 19:21:42.085535 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:42.085536    4372 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d72beb/agc-cred-919331a7b490: secret "agc-cred-919331a7b490" not found
Oct 30 19:21:42.085735 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:42.085545    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl: [secret "agc-cred-1a4160222042" not found, secret "agc-cred-919331a7b490" not found]
Oct 30 19:21:42.085735 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:42.085592    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-provider-credentials podName:88adeb15-521e-4bf1-85bd-6b549ea5c460 nodeName:}" failed. No retries permitted until 2026-10-30 19:21:46.085577625 +0000 UTC m=+2919.953995933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl" (UID: "88adeb15-521e-4bf1-85bd-6b549ea5c460") : [secret "agc-cred-1a4160222042" not found, secret "agc-cred-919331a7b490" not found]
Oct 30 19:21:42.265688 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:42.265655    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56" path="/var/lib/kubelet/pods/2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56/volumes"
Oct 30 19:21:42.388280 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:42.388212    4372 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-plugins-e2e-extmod-b-d72beb: configmap "payload-processing-external-model-plugins-e2e-extmod-b-d72beb" not found
Oct 30 19:21:42.388280 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:42.388281    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-plugins-config-volume podName:b8447ca7-004b-480f-8d70-c1896a5aa6ca nodeName:}" failed. No retries permitted until 2026-10-30 19:21:44.388266672 +0000 UTC m=+2918.256684997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84" (UID: "b8447ca7-004b-480f-8d70-c1896a5aa6ca") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-d72beb" not found
Oct 30 19:21:42.878929 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:42.878888    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" containerID="cri-o://46bc168ea2612f86f256fe3d7440fd6eb3acee8f7a167aa5ccfca5715c9a2b02" gracePeriod=45
Oct 30 19:21:42.939416 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:42.939379    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": dial tcp 10.133.0.62:8000: connect: connection refused"
Oct 30 19:21:43.155956 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.155927    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 19:21:43.322520 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.322480    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-model-cache\") pod \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") "
Oct 30 19:21:43.322724 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.322549    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-dshm\") pod \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") "
Oct 30 19:21:43.322724 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.322585    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-home\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-home\") pod \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") "
Oct 30 19:21:43.322874 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.322736    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03a3b3a8-b37f-4c86-99fa-4e4494987be7-tls-certs\" (UniqueName: \"kubernetes.io/secret/03a3b3a8-b37f-4c86-99fa-4e4494987be7-tls-certs\") pod \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") "
Oct 30 19:21:43.322874 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.322773    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-tmp-dir\") pod \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") "
Oct 30 19:21:43.322874 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.322785    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-model-cache" pod "03a3b3a8-b37f-4c86-99fa-4e4494987be7" (UID: "03a3b3a8-b37f-4c86-99fa-4e4494987be7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.322874 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.322826    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a3b3a8-b37f-4c86-99fa-4e4494987be7-kube-api-access-h29md\" (UniqueName: \"kubernetes.io/projected/03a3b3a8-b37f-4c86-99fa-4e4494987be7-kube-api-access-h29md\") pod \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\" (UID: \"03a3b3a8-b37f-4c86-99fa-4e4494987be7\") "
Oct 30 19:21:43.323100 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.322944    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-home" pod "03a3b3a8-b37f-4c86-99fa-4e4494987be7" (UID: "03a3b3a8-b37f-4c86-99fa-4e4494987be7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.323500 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.323474    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-tmp-dir" pod "03a3b3a8-b37f-4c86-99fa-4e4494987be7" (UID: "03a3b3a8-b37f-4c86-99fa-4e4494987be7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.323681 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.323557    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.323850 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.323696    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.325932 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.325906    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a3b3a8-b37f-4c86-99fa-4e4494987be7-tls-certs" pod "03a3b3a8-b37f-4c86-99fa-4e4494987be7" (UID: "03a3b3a8-b37f-4c86-99fa-4e4494987be7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:43.326016 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.325958    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a3b3a8-b37f-4c86-99fa-4e4494987be7-kube-api-access-h29md" pod "03a3b3a8-b37f-4c86-99fa-4e4494987be7" (UID: "03a3b3a8-b37f-4c86-99fa-4e4494987be7"). InnerVolumeSpecName "kube-api-access-h29md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:43.327290 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.327266    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-dshm" pod "03a3b3a8-b37f-4c86-99fa-4e4494987be7" (UID: "03a3b3a8-b37f-4c86-99fa-4e4494987be7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:43.424795 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.424714    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.424795 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.424744    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03a3b3a8-b37f-4c86-99fa-4e4494987be7-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.424795 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.424753    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a3b3a8-b37f-4c86-99fa-4e4494987be7-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.424795 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.424763    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h29md\" (UniqueName: \"kubernetes.io/projected/03a3b3a8-b37f-4c86-99fa-4e4494987be7-kube-api-access-h29md\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:43.526333 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:43.526299    4372 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-plugins-e2e-extmod-a-21d896: configmap "payload-processing-external-model-plugins-e2e-extmod-a-21d896" not found
Oct 30 19:21:43.526493 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:43.526379    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-plugins-config-volume podName:e86c2cb2-8ae4-4a42-b277-3779c70b995a nodeName:}" failed. No retries permitted until 2026-10-30 19:21:47.526361392 +0000 UTC m=+2921.394779714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86" (UID: "e86c2cb2-8ae4-4a42-b277-3779c70b995a") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-21d896" not found
Oct 30 19:21:43.526493 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:43.526395    4372 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-21d896/agc-cred-676f802b0d13: secret "agc-cred-676f802b0d13" not found
Oct 30 19:21:43.526493 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:43.526416    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86: secret "agc-cred-676f802b0d13" not found
Oct 30 19:21:43.526493 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:43.526452    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-provider-credentials podName:e86c2cb2-8ae4-4a42-b277-3779c70b995a nodeName:}" failed. No retries permitted until 2026-10-30 19:21:47.526441049 +0000 UTC m=+2921.394859357 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86" (UID: "e86c2cb2-8ae4-4a42-b277-3779c70b995a") : secret "agc-cred-676f802b0d13" not found
Oct 30 19:21:43.526493 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:43.526396    4372 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d72beb/agc-cred-1a4160222042: secret "agc-cred-1a4160222042" not found
Oct 30 19:21:43.526493 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:43.526467    4372 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84: secret "agc-cred-1a4160222042" not found
Oct 30 19:21:43.526493 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:43.526487    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-provider-credentials podName:b8447ca7-004b-480f-8d70-c1896a5aa6ca nodeName:}" failed. No retries permitted until 2026-10-30 19:21:47.526480095 +0000 UTC m=+2921.394898403 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84" (UID: "b8447ca7-004b-480f-8d70-c1896a5aa6ca") : secret "agc-cred-1a4160222042" not found
Oct 30 19:21:43.843180 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.843142    4372 generic.go:356] "Generic (PLEG): container finished" podID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerID="46bc168ea2612f86f256fe3d7440fd6eb3acee8f7a167aa5ccfca5715c9a2b02" exitCode=0
Oct 30 19:21:43.843345 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.843209    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"
Oct 30 19:21:43.843345 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.843214    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" event={"ID":"03a3b3a8-b37f-4c86-99fa-4e4494987be7","Type":"ContainerDied","Data":"46bc168ea2612f86f256fe3d7440fd6eb3acee8f7a167aa5ccfca5715c9a2b02"}
Oct 30 19:21:43.843345 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.843263    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb" event={"ID":"03a3b3a8-b37f-4c86-99fa-4e4494987be7","Type":"ContainerDied","Data":"d0523b2b5b4abf05943571932fa79387f57880b02035e14740c8c02cea1ee8bb"}
Oct 30 19:21:43.843345 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.843281    4372 scope.go:122] "RemoveContainer" containerID="46bc168ea2612f86f256fe3d7440fd6eb3acee8f7a167aa5ccfca5715c9a2b02"
Oct 30 19:21:43.854977 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.854959    4372 scope.go:122] "RemoveContainer" containerID="46bc168ea2612f86f256fe3d7440fd6eb3acee8f7a167aa5ccfca5715c9a2b02"
Oct 30 19:21:43.855242 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:43.855223    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46bc168ea2612f86f256fe3d7440fd6eb3acee8f7a167aa5ccfca5715c9a2b02\": container with ID starting with 46bc168ea2612f86f256fe3d7440fd6eb3acee8f7a167aa5ccfca5715c9a2b02 not found: ID does not exist" containerID="46bc168ea2612f86f256fe3d7440fd6eb3acee8f7a167aa5ccfca5715c9a2b02"
Oct 30 19:21:43.855344 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.855249    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bc168ea2612f86f256fe3d7440fd6eb3acee8f7a167aa5ccfca5715c9a2b02"} err="failed to get container status \"46bc168ea2612f86f256fe3d7440fd6eb3acee8f7a167aa5ccfca5715c9a2b02\": rpc error: code = NotFound desc = could not find container \"46bc168ea2612f86f256fe3d7440fd6eb3acee8f7a167aa5ccfca5715c9a2b02\": container with ID starting with 46bc168ea2612f86f256fe3d7440fd6eb3acee8f7a167aa5ccfca5715c9a2b02 not found: ID does not exist"
Oct 30 19:21:43.873674 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.873648    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"]
Oct 30 19:21:43.876818 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:43.876793    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-da70a5/distinct-w4-da70a5-kserve-54f75954bc-hdgpb"]
Oct 30 19:21:44.267181 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:44.267086    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" path="/var/lib/kubelet/pods/03a3b3a8-b37f-4c86-99fa-4e4494987be7/volumes"
Oct 30 19:21:44.438589 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:44.438557    4372 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-plugins-e2e-extmod-b-d72beb: configmap "payload-processing-external-model-plugins-e2e-extmod-b-d72beb" not found
Oct 30 19:21:44.438732 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:44.438619    4372 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-plugins-config-volume podName:b8447ca7-004b-480f-8d70-c1896a5aa6ca nodeName:}" failed. No retries permitted until 2026-10-30 19:21:48.438605444 +0000 UTC m=+2922.307023755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84" (UID: "b8447ca7-004b-480f-8d70-c1896a5aa6ca") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-d72beb" not found
Oct 30 19:21:44.994404 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:44.994361    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl" podUID="88adeb15-521e-4bf1-85bd-6b549ea5c460" containerName="payload-processing" containerID="cri-o://6d9248dc3acd972750a037253e9985f457c15fb18c164fccfd4b2eab3f3ecbed" gracePeriod=25
Oct 30 19:21:45.260234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.260173    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:21:45.291914 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.291879    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj" podUID="31d23f77-b321-4ed8-b1e2-e0d1376d5697" containerName="payload-processing" containerID="cri-o://f39689a3d7ca0e8f1404c4dbddaf919c832160c01853e7f69cf93c0b8491bb08" gracePeriod=25
Oct 30 19:21:45.305322 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.305288    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg" podUID="925f7b63-9cb5-4ee8-8867-91530abaf46c" containerName="payload-pre-processing" containerID="cri-o://6a23df6fdc6c45da962a2f48b6ea1c831a4bc3eab426affc3766c7a98e26e256" gracePeriod=25
Oct 30 19:21:45.350522 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.350497    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88adeb15-521e-4bf1-85bd-6b549ea5c460-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88adeb15-521e-4bf1-85bd-6b549ea5c460-plugins-config-volume\") pod \"88adeb15-521e-4bf1-85bd-6b549ea5c460\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") "
Oct 30 19:21:45.350641 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.350614    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88adeb15-521e-4bf1-85bd-6b549ea5c460-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88adeb15-521e-4bf1-85bd-6b549ea5c460-routing-overlay-volume\") pod \"88adeb15-521e-4bf1-85bd-6b549ea5c460\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") "
Oct 30 19:21:45.350750 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.350735    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-kube-api-access-zwtfq\" (UniqueName: \"kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-kube-api-access-zwtfq\") pod \"88adeb15-521e-4bf1-85bd-6b549ea5c460\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") "
Oct 30 19:21:45.350816 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.350772    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-provider-credentials\" (UniqueName: \"kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-provider-credentials\") pod \"88adeb15-521e-4bf1-85bd-6b549ea5c460\" (UID: \"88adeb15-521e-4bf1-85bd-6b549ea5c460\") "
Oct 30 19:21:45.350974 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.350950    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88adeb15-521e-4bf1-85bd-6b549ea5c460-plugins-config-volume" pod "88adeb15-521e-4bf1-85bd-6b549ea5c460" (UID: "88adeb15-521e-4bf1-85bd-6b549ea5c460"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:45.351058 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.350962    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88adeb15-521e-4bf1-85bd-6b549ea5c460-routing-overlay-volume" pod "88adeb15-521e-4bf1-85bd-6b549ea5c460" (UID: "88adeb15-521e-4bf1-85bd-6b549ea5c460"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:45.351341 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.351321    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88adeb15-521e-4bf1-85bd-6b549ea5c460-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:45.351422 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.351345    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88adeb15-521e-4bf1-85bd-6b549ea5c460-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:45.353034 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.353010    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-provider-credentials" pod "88adeb15-521e-4bf1-85bd-6b549ea5c460" (UID: "88adeb15-521e-4bf1-85bd-6b549ea5c460"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:45.353094 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.353031    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-kube-api-access-zwtfq" pod "88adeb15-521e-4bf1-85bd-6b549ea5c460" (UID: "88adeb15-521e-4bf1-85bd-6b549ea5c460"). InnerVolumeSpecName "kube-api-access-zwtfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:45.452699 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.452671    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwtfq\" (UniqueName: \"kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-kube-api-access-zwtfq\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:45.452699 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.452697    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88adeb15-521e-4bf1-85bd-6b549ea5c460-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:45.820305 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.820263    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86" podUID="e86c2cb2-8ae4-4a42-b277-3779c70b995a" containerName="payload-processing" containerID="cri-o://29572d3366e4f668d99664efd31e11dd1480a48027b60467e62db7f449cc571c" gracePeriod=25
Oct 30 19:21:45.859679 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.859646    4372 generic.go:356] "Generic (PLEG): container finished" podID="88adeb15-521e-4bf1-85bd-6b549ea5c460" containerID="6d9248dc3acd972750a037253e9985f457c15fb18c164fccfd4b2eab3f3ecbed" exitCode=0
Oct 30 19:21:45.859815 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.859706    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"
Oct 30 19:21:45.859815 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.859721    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl" event={"ID":"88adeb15-521e-4bf1-85bd-6b549ea5c460","Type":"ContainerDied","Data":"6d9248dc3acd972750a037253e9985f457c15fb18c164fccfd4b2eab3f3ecbed"}
Oct 30 19:21:45.859815 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.859775    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl" event={"ID":"88adeb15-521e-4bf1-85bd-6b549ea5c460","Type":"ContainerDied","Data":"0f9fee5488c327bdeb4b3543b9935d5c438bdb67b67fa8151956ddca7c4fd8dc"}
Oct 30 19:21:45.859815 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.859797    4372 scope.go:122] "RemoveContainer" containerID="6d9248dc3acd972750a037253e9985f457c15fb18c164fccfd4b2eab3f3ecbed"
Oct 30 19:21:45.902804 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.902776    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"]
Oct 30 19:21:45.907136 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.907100    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-57bclz6dl"]
Oct 30 19:21:45.932486 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.932465    4372 scope.go:122] "RemoveContainer" containerID="6d9248dc3acd972750a037253e9985f457c15fb18c164fccfd4b2eab3f3ecbed"
Oct 30 19:21:45.932823 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:45.932802    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9248dc3acd972750a037253e9985f457c15fb18c164fccfd4b2eab3f3ecbed\": container with ID starting with 6d9248dc3acd972750a037253e9985f457c15fb18c164fccfd4b2eab3f3ecbed not found: ID does not exist" containerID="6d9248dc3acd972750a037253e9985f457c15fb18c164fccfd4b2eab3f3ecbed"
Oct 30 19:21:45.932886 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:45.932832    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9248dc3acd972750a037253e9985f457c15fb18c164fccfd4b2eab3f3ecbed"} err="failed to get container status \"6d9248dc3acd972750a037253e9985f457c15fb18c164fccfd4b2eab3f3ecbed\": rpc error: code = NotFound desc = could not find container \"6d9248dc3acd972750a037253e9985f457c15fb18c164fccfd4b2eab3f3ecbed\": container with ID starting with 6d9248dc3acd972750a037253e9985f457c15fb18c164fccfd4b2eab3f3ecbed not found: ID does not exist"
Oct 30 19:21:46.100042 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.100019    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:46.264423 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.264397    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-routing-overlay-volume\") pod \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") "
Oct 30 19:21:46.264604 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.264453    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-plugins-config-volume\") pod \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") "
Oct 30 19:21:46.264604 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.264484    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-provider-credentials\") pod \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") "
Oct 30 19:21:46.264604 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.264526    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-kube-api-access-4xd47\" (UniqueName: \"kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-kube-api-access-4xd47\") pod \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\" (UID: \"e86c2cb2-8ae4-4a42-b277-3779c70b995a\") "
Oct 30 19:21:46.264839 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.264808    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-routing-overlay-volume" pod "e86c2cb2-8ae4-4a42-b277-3779c70b995a" (UID: "e86c2cb2-8ae4-4a42-b277-3779c70b995a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:46.264918 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.264830    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-plugins-config-volume" pod "e86c2cb2-8ae4-4a42-b277-3779c70b995a" (UID: "e86c2cb2-8ae4-4a42-b277-3779c70b995a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:46.265116 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.265097    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:46.265219 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.265152    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e86c2cb2-8ae4-4a42-b277-3779c70b995a-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:46.266051 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.266017    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88adeb15-521e-4bf1-85bd-6b549ea5c460" path="/var/lib/kubelet/pods/88adeb15-521e-4bf1-85bd-6b549ea5c460/volumes"
Oct 30 19:21:46.266929 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.266910    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-provider-credentials" pod "e86c2cb2-8ae4-4a42-b277-3779c70b995a" (UID: "e86c2cb2-8ae4-4a42-b277-3779c70b995a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:46.267013 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.266994    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-kube-api-access-4xd47" pod "e86c2cb2-8ae4-4a42-b277-3779c70b995a" (UID: "e86c2cb2-8ae4-4a42-b277-3779c70b995a"). InnerVolumeSpecName "kube-api-access-4xd47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:46.365932 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.365903    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:46.366287 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.365933    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xd47\" (UniqueName: \"kubernetes.io/projected/e86c2cb2-8ae4-4a42-b277-3779c70b995a-kube-api-access-4xd47\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:46.827854 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.827812    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84" podUID="b8447ca7-004b-480f-8d70-c1896a5aa6ca" containerName="payload-processing" containerID="cri-o://f794317bdc56dfd2b32e16864ab0a6bdeeb28bd6685fa530fcd7ac9285a709a4" gracePeriod=25
Oct 30 19:21:46.867223 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.867195    4372 generic.go:356] "Generic (PLEG): container finished" podID="e86c2cb2-8ae4-4a42-b277-3779c70b995a" containerID="29572d3366e4f668d99664efd31e11dd1480a48027b60467e62db7f449cc571c" exitCode=0
Oct 30 19:21:46.867348 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.867244    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"
Oct 30 19:21:46.867348 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.867272    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86" event={"ID":"e86c2cb2-8ae4-4a42-b277-3779c70b995a","Type":"ContainerDied","Data":"29572d3366e4f668d99664efd31e11dd1480a48027b60467e62db7f449cc571c"}
Oct 30 19:21:46.867348 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.867329    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86" event={"ID":"e86c2cb2-8ae4-4a42-b277-3779c70b995a","Type":"ContainerDied","Data":"84ed9bae7e0815acc9f09fb09492da3aa2becc557ba32dac703b1801b2a3214a"}
Oct 30 19:21:46.867518 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.867352    4372 scope.go:122] "RemoveContainer" containerID="29572d3366e4f668d99664efd31e11dd1480a48027b60467e62db7f449cc571c"
Oct 30 19:21:46.911420 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.911392    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"]
Oct 30 19:21:46.919431 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.919402    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-7d8f5tq86"]
Oct 30 19:21:46.940729 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.940707    4372 scope.go:122] "RemoveContainer" containerID="29572d3366e4f668d99664efd31e11dd1480a48027b60467e62db7f449cc571c"
Oct 30 19:21:46.941029 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:46.941007    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29572d3366e4f668d99664efd31e11dd1480a48027b60467e62db7f449cc571c\": container with ID starting with 29572d3366e4f668d99664efd31e11dd1480a48027b60467e62db7f449cc571c not found: ID does not exist" containerID="29572d3366e4f668d99664efd31e11dd1480a48027b60467e62db7f449cc571c"
Oct 30 19:21:46.941104 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:46.941040    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29572d3366e4f668d99664efd31e11dd1480a48027b60467e62db7f449cc571c"} err="failed to get container status \"29572d3366e4f668d99664efd31e11dd1480a48027b60467e62db7f449cc571c\": rpc error: code = NotFound desc = could not find container \"29572d3366e4f668d99664efd31e11dd1480a48027b60467e62db7f449cc571c\": container with ID starting with 29572d3366e4f668d99664efd31e11dd1480a48027b60467e62db7f449cc571c not found: ID does not exist"
Oct 30 19:21:47.089721 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.089660    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:47.176569 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.176541    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-routing-overlay-volume\") pod \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") "
Oct 30 19:21:47.176763 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.176617    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-kube-api-access-r9nhn\" (UniqueName: \"kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-kube-api-access-r9nhn\") pod \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") "
Oct 30 19:21:47.176763 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.176652    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-provider-credentials\") pod \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") "
Oct 30 19:21:47.176894 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.176868    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-plugins-config-volume\") pod \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\" (UID: \"b8447ca7-004b-480f-8d70-c1896a5aa6ca\") "
Oct 30 19:21:47.176983 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.176947    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-routing-overlay-volume" pod "b8447ca7-004b-480f-8d70-c1896a5aa6ca" (UID: "b8447ca7-004b-480f-8d70-c1896a5aa6ca"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:47.177376 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.177350    4372 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-routing-overlay-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:47.177376 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.177347    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-plugins-config-volume" pod "b8447ca7-004b-480f-8d70-c1896a5aa6ca" (UID: "b8447ca7-004b-480f-8d70-c1896a5aa6ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:47.179688 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.179658    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-provider-credentials" pod "b8447ca7-004b-480f-8d70-c1896a5aa6ca" (UID: "b8447ca7-004b-480f-8d70-c1896a5aa6ca"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:47.179790 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.179745    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-kube-api-access-r9nhn" pod "b8447ca7-004b-480f-8d70-c1896a5aa6ca" (UID: "b8447ca7-004b-480f-8d70-c1896a5aa6ca"). InnerVolumeSpecName "kube-api-access-r9nhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:47.278426 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.278401    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8447ca7-004b-480f-8d70-c1896a5aa6ca-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:47.278426 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.278425    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9nhn\" (UniqueName: \"kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-kube-api-access-r9nhn\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:47.278555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.278434    4372 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8447ca7-004b-480f-8d70-c1896a5aa6ca-provider-credentials\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:47.435220 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.434936    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"]
Oct 30 19:21:47.435709 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.435281    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn" podUID="adfcf9c2-e42d-443c-ae5c-d734dbd4bff7" containerName="maas-api" containerID="cri-o://faf45de0c81f6c4f1261e3b6a5421f841ade7efa28ebe0d2615564c4522121cc" gracePeriod=30
Oct 30 19:21:47.700250 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.700226    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 19:21:47.787140 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.787092    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-openshift-service-ca\") pod \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") "
Oct 30 19:21:47.787331 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.787292    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-kube-api-access-db876\" (UniqueName: \"kubernetes.io/projected/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-kube-api-access-db876\") pod \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") "
Oct 30 19:21:47.787432 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.787408    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-maas-api-tls\") pod \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") "
Oct 30 19:21:47.787579 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.787444    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-metrics-certs\") pod \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\" (UID: \"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7\") "
Oct 30 19:21:47.787772 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.787744    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-openshift-service-ca" pod "adfcf9c2-e42d-443c-ae5c-d734dbd4bff7" (UID: "adfcf9c2-e42d-443c-ae5c-d734dbd4bff7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:47.787948 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.787927    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:47.790191 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.790117    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-maas-api-tls" pod "adfcf9c2-e42d-443c-ae5c-d734dbd4bff7" (UID: "adfcf9c2-e42d-443c-ae5c-d734dbd4bff7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:47.790679 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.790656    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-metrics-certs" pod "adfcf9c2-e42d-443c-ae5c-d734dbd4bff7" (UID: "adfcf9c2-e42d-443c-ae5c-d734dbd4bff7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:47.791084 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.791067    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-kube-api-access-db876" pod "adfcf9c2-e42d-443c-ae5c-d734dbd4bff7" (UID: "adfcf9c2-e42d-443c-ae5c-d734dbd4bff7"). InnerVolumeSpecName "kube-api-access-db876". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:47.893151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.882579    4372 generic.go:356] "Generic (PLEG): container finished" podID="adfcf9c2-e42d-443c-ae5c-d734dbd4bff7" containerID="faf45de0c81f6c4f1261e3b6a5421f841ade7efa28ebe0d2615564c4522121cc" exitCode=0
Oct 30 19:21:47.893151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.883261    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn" event={"ID":"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7","Type":"ContainerDied","Data":"faf45de0c81f6c4f1261e3b6a5421f841ade7efa28ebe0d2615564c4522121cc"}
Oct 30 19:21:47.893151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.883362    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn" event={"ID":"adfcf9c2-e42d-443c-ae5c-d734dbd4bff7","Type":"ContainerDied","Data":"ae0746a5527c3a956273d910708a4e90a68bd5c298ec1f5dd62d2c6e642b9aa0"}
Oct 30 19:21:47.893151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.883422    4372 scope.go:122] "RemoveContainer" containerID="faf45de0c81f6c4f1261e3b6a5421f841ade7efa28ebe0d2615564c4522121cc"
Oct 30 19:21:47.893151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.883597    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"
Oct 30 19:21:47.893151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.890700    4372 generic.go:356] "Generic (PLEG): container finished" podID="b8447ca7-004b-480f-8d70-c1896a5aa6ca" containerID="f794317bdc56dfd2b32e16864ab0a6bdeeb28bd6685fa530fcd7ac9285a709a4" exitCode=0
Oct 30 19:21:47.893151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.890799    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84" event={"ID":"b8447ca7-004b-480f-8d70-c1896a5aa6ca","Type":"ContainerDied","Data":"f794317bdc56dfd2b32e16864ab0a6bdeeb28bd6685fa530fcd7ac9285a709a4"}
Oct 30 19:21:47.893151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.890841    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84" event={"ID":"b8447ca7-004b-480f-8d70-c1896a5aa6ca","Type":"ContainerDied","Data":"923ec4f8e90c5a612007c0e7aa0ac424e4fc8d7dee09070245aafed4c882933f"}
Oct 30 19:21:47.893151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.890917    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"
Oct 30 19:21:47.903316 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.896488    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:47.903316 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.896521    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:47.903316 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.896538    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db876\" (UniqueName: \"kubernetes.io/projected/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7-kube-api-access-db876\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:47.908532 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.908494    4372 scope.go:122] "RemoveContainer" containerID="faf45de0c81f6c4f1261e3b6a5421f841ade7efa28ebe0d2615564c4522121cc"
Oct 30 19:21:47.919971 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:47.919934    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faf45de0c81f6c4f1261e3b6a5421f841ade7efa28ebe0d2615564c4522121cc\": container with ID starting with faf45de0c81f6c4f1261e3b6a5421f841ade7efa28ebe0d2615564c4522121cc not found: ID does not exist" containerID="faf45de0c81f6c4f1261e3b6a5421f841ade7efa28ebe0d2615564c4522121cc"
Oct 30 19:21:47.920098 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.919981    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faf45de0c81f6c4f1261e3b6a5421f841ade7efa28ebe0d2615564c4522121cc"} err="failed to get container status \"faf45de0c81f6c4f1261e3b6a5421f841ade7efa28ebe0d2615564c4522121cc\": rpc error: code = NotFound desc = could not find container \"faf45de0c81f6c4f1261e3b6a5421f841ade7efa28ebe0d2615564c4522121cc\": container with ID starting with faf45de0c81f6c4f1261e3b6a5421f841ade7efa28ebe0d2615564c4522121cc not found: ID does not exist"
Oct 30 19:21:47.920098 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.920005    4372 scope.go:122] "RemoveContainer" containerID="f794317bdc56dfd2b32e16864ab0a6bdeeb28bd6685fa530fcd7ac9285a709a4"
Oct 30 19:21:47.952322 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.952260    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"]
Oct 30 19:21:47.955280 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.955252    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d7870-7664765588-rnxgn"]
Oct 30 19:21:47.956821 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.956801    4372 scope.go:122] "RemoveContainer" containerID="f794317bdc56dfd2b32e16864ab0a6bdeeb28bd6685fa530fcd7ac9285a709a4"
Oct 30 19:21:47.957108 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:47.957082    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f794317bdc56dfd2b32e16864ab0a6bdeeb28bd6685fa530fcd7ac9285a709a4\": container with ID starting with f794317bdc56dfd2b32e16864ab0a6bdeeb28bd6685fa530fcd7ac9285a709a4 not found: ID does not exist" containerID="f794317bdc56dfd2b32e16864ab0a6bdeeb28bd6685fa530fcd7ac9285a709a4"
Oct 30 19:21:47.957205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.957121    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f794317bdc56dfd2b32e16864ab0a6bdeeb28bd6685fa530fcd7ac9285a709a4"} err="failed to get container status \"f794317bdc56dfd2b32e16864ab0a6bdeeb28bd6685fa530fcd7ac9285a709a4\": rpc error: code = NotFound desc = could not find container \"f794317bdc56dfd2b32e16864ab0a6bdeeb28bd6685fa530fcd7ac9285a709a4\": container with ID starting with f794317bdc56dfd2b32e16864ab0a6bdeeb28bd6685fa530fcd7ac9285a709a4 not found: ID does not exist"
Oct 30 19:21:47.970010 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.966905    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"]
Oct 30 19:21:48.008154 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:47.995397    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d72beb/payload-processing-external-model-e2e-extmod-b-d72beb-7c65qdl84"]
Oct 30 19:21:48.270491 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:48.270399    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adfcf9c2-e42d-443c-ae5c-d734dbd4bff7" path="/var/lib/kubelet/pods/adfcf9c2-e42d-443c-ae5c-d734dbd4bff7/volumes"
Oct 30 19:21:48.270963 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:48.270939    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8447ca7-004b-480f-8d70-c1896a5aa6ca" path="/var/lib/kubelet/pods/b8447ca7-004b-480f-8d70-c1896a5aa6ca/volumes"
Oct 30 19:21:48.271446 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:48.271425    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e86c2cb2-8ae4-4a42-b277-3779c70b995a" path="/var/lib/kubelet/pods/e86c2cb2-8ae4-4a42-b277-3779c70b995a/volumes"
Oct 30 19:21:50.824921 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:50.824889    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"]
Oct 30 19:21:50.825512 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:50.825088    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq" podUID="93367dd2-e078-4c5f-9e84-e8e5832bb87f" containerName="istio-proxy" containerID="cri-o://498e0226236c63e590deab53af930a68805a63adfe4d7057c31e91d4d5c4d9c9" gracePeriod=30
Oct 30 19:21:50.827922 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:50.827898    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"]
Oct 30 19:21:50.828119 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:50.828095    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm" podUID="9fafb996-e969-4c99-be6b-421487aed44e" containerName="istio-proxy" containerID="cri-o://64d8cfc1735e89a188a9a10309d67aafa966779130826b2e631db3df3581639d" gracePeriod=30
Oct 30 19:21:52.515875 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:52.515834    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg" podUID="925f7b63-9cb5-4ee8-8867-91530abaf46c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:52.515875 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:52.515844    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj" podUID="31d23f77-b321-4ed8-b1e2-e0d1376d5697" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:55.998166 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:55.997946    4372 generic.go:356] "Generic (PLEG): container finished" podID="9fafb996-e969-4c99-be6b-421487aed44e" containerID="64d8cfc1735e89a188a9a10309d67aafa966779130826b2e631db3df3581639d" exitCode=0
Oct 30 19:21:55.998166 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:55.998065    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm" event={"ID":"9fafb996-e969-4c99-be6b-421487aed44e","Type":"ContainerDied","Data":"64d8cfc1735e89a188a9a10309d67aafa966779130826b2e631db3df3581639d"}
Oct 30 19:21:56.000975 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.000944    4372 generic.go:356] "Generic (PLEG): container finished" podID="93367dd2-e078-4c5f-9e84-e8e5832bb87f" containerID="498e0226236c63e590deab53af930a68805a63adfe4d7057c31e91d4d5c4d9c9" exitCode=0
Oct 30 19:21:56.001167 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.000996    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq" event={"ID":"93367dd2-e078-4c5f-9e84-e8e5832bb87f","Type":"ContainerDied","Data":"498e0226236c63e590deab53af930a68805a63adfe4d7057c31e91d4d5c4d9c9"}
Oct 30 19:21:56.126781 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.126756    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 19:21:56.172695 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.172671    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 19:21:56.217399 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217366    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-podinfo\") pod \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") "
Oct 30 19:21:56.217576 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217450    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-istio-data\") pod \"9fafb996-e969-4c99-be6b-421487aed44e\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") "
Oct 30 19:21:56.217576 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217482    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-workload-certs\") pod \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") "
Oct 30 19:21:56.217576 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217532    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fafb996-e969-4c99-be6b-421487aed44e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fafb996-e969-4c99-be6b-421487aed44e-istiod-ca-cert\") pod \"9fafb996-e969-4c99-be6b-421487aed44e\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") "
Oct 30 19:21:56.217576 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217561    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-workload-socket\") pod \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") "
Oct 30 19:21:56.217837 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217608    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fafb996-e969-4c99-be6b-421487aed44e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fafb996-e969-4c99-be6b-421487aed44e-istio-podinfo\") pod \"9fafb996-e969-4c99-be6b-421487aed44e\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") "
Oct 30 19:21:56.217837 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217639    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-credential-socket\") pod \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") "
Oct 30 19:21:56.217837 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217665    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-credential-socket\") pod \"9fafb996-e969-4c99-be6b-421487aed44e\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") "
Oct 30 19:21:56.217837 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217730    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-istio-data" pod "9fafb996-e969-4c99-be6b-421487aed44e" (UID: "9fafb996-e969-4c99-be6b-421487aed44e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:56.217837 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217755    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-envoy\") pod \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") "
Oct 30 19:21:56.217837 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217786    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-token\" (UniqueName: \"kubernetes.io/projected/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-token\") pod \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") "
Oct 30 19:21:56.218249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217849    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-workload-certs\") pod \"9fafb996-e969-4c99-be6b-421487aed44e\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") "
Oct 30 19:21:56.218249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217854    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-workload-certs" pod "93367dd2-e078-4c5f-9e84-e8e5832bb87f" (UID: "93367dd2-e078-4c5f-9e84-e8e5832bb87f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:56.218249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217889    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fafb996-e969-4c99-be6b-421487aed44e-kube-api-access-97mqt\" (UniqueName: \"kubernetes.io/projected/9fafb996-e969-4c99-be6b-421487aed44e-kube-api-access-97mqt\") pod \"9fafb996-e969-4c99-be6b-421487aed44e\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") "
Oct 30 19:21:56.218249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217933    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93367dd2-e078-4c5f-9e84-e8e5832bb87f-kube-api-access-fm2ws\" (UniqueName: \"kubernetes.io/projected/93367dd2-e078-4c5f-9e84-e8e5832bb87f-kube-api-access-fm2ws\") pod \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") "
Oct 30 19:21:56.218249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217965    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-workload-socket" pod "93367dd2-e078-4c5f-9e84-e8e5832bb87f" (UID: "93367dd2-e078-4c5f-9e84-e8e5832bb87f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:56.218249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.217995    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-istio-envoy\") pod \"9fafb996-e969-4c99-be6b-421487aed44e\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") "
Oct 30 19:21:56.218249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.218012    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-credential-socket" pod "9fafb996-e969-4c99-be6b-421487aed44e" (UID: "9fafb996-e969-4c99-be6b-421487aed44e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:56.218249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.218021    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fafb996-e969-4c99-be6b-421487aed44e-istio-token\" (UniqueName: \"kubernetes.io/projected/9fafb996-e969-4c99-be6b-421487aed44e-istio-token\") pod \"9fafb996-e969-4c99-be6b-421487aed44e\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") "
Oct 30 19:21:56.218249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.218061    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-workload-socket\") pod \"9fafb996-e969-4c99-be6b-421487aed44e\" (UID: \"9fafb996-e969-4c99-be6b-421487aed44e\") "
Oct 30 19:21:56.218249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.218113    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-credential-socket" pod "93367dd2-e078-4c5f-9e84-e8e5832bb87f" (UID: "93367dd2-e078-4c5f-9e84-e8e5832bb87f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:56.218249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.218247    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fafb996-e969-4c99-be6b-421487aed44e-istiod-ca-cert" pod "9fafb996-e969-4c99-be6b-421487aed44e" (UID: "9fafb996-e969-4c99-be6b-421487aed44e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:56.219053 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.218941    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-workload-socket" pod "9fafb996-e969-4c99-be6b-421487aed44e" (UID: "9fafb996-e969-4c99-be6b-421487aed44e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:56.219289 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.219263    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-workload-certs" pod "9fafb996-e969-4c99-be6b-421487aed44e" (UID: "9fafb996-e969-4c99-be6b-421487aed44e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:56.221581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.221551    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-podinfo" pod "93367dd2-e078-4c5f-9e84-e8e5832bb87f" (UID: "93367dd2-e078-4c5f-9e84-e8e5832bb87f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:21:56.221707 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.221589    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fafb996-e969-4c99-be6b-421487aed44e-istio-podinfo" pod "9fafb996-e969-4c99-be6b-421487aed44e" (UID: "9fafb996-e969-4c99-be6b-421487aed44e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:21:56.221707 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.221606    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-data\") pod \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") "
Oct 30 19:21:56.221707 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.221680    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istiod-ca-cert\") pod \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\" (UID: \"93367dd2-e078-4c5f-9e84-e8e5832bb87f\") "
Oct 30 19:21:56.221911 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.221609    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-token" pod "93367dd2-e078-4c5f-9e84-e8e5832bb87f" (UID: "93367dd2-e078-4c5f-9e84-e8e5832bb87f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:56.221911 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.221774    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-istio-envoy" pod "9fafb996-e969-4c99-be6b-421487aed44e" (UID: "9fafb996-e969-4c99-be6b-421487aed44e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:56.222033 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.221836    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-data" pod "93367dd2-e078-4c5f-9e84-e8e5832bb87f" (UID: "93367dd2-e078-4c5f-9e84-e8e5832bb87f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:56.222033 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222017    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istiod-ca-cert" pod "93367dd2-e078-4c5f-9e84-e8e5832bb87f" (UID: "93367dd2-e078-4c5f-9e84-e8e5832bb87f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:56.222142 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222034    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93367dd2-e078-4c5f-9e84-e8e5832bb87f-kube-api-access-fm2ws" pod "93367dd2-e078-4c5f-9e84-e8e5832bb87f" (UID: "93367dd2-e078-4c5f-9e84-e8e5832bb87f"). InnerVolumeSpecName "kube-api-access-fm2ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:56.222142 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222063    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fafb996-e969-4c99-be6b-421487aed44e-istio-token" pod "9fafb996-e969-4c99-be6b-421487aed44e" (UID: "9fafb996-e969-4c99-be6b-421487aed44e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:56.222340 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222321    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222380 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222347    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222380 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222362    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm2ws\" (UniqueName: \"kubernetes.io/projected/93367dd2-e078-4c5f-9e84-e8e5832bb87f-kube-api-access-fm2ws\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222446 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222377    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222446 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222390    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fafb996-e969-4c99-be6b-421487aed44e-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222446 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222404    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222446 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222418    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222446 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222430    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222445    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222458    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222472    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222485    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fafb996-e969-4c99-be6b-421487aed44e-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222498    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222511    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fafb996-e969-4c99-be6b-421487aed44e-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222529    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222635 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222556    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fafb996-e969-4c99-be6b-421487aed44e-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.222996 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.222973    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fafb996-e969-4c99-be6b-421487aed44e-kube-api-access-97mqt" pod "9fafb996-e969-4c99-be6b-421487aed44e" (UID: "9fafb996-e969-4c99-be6b-421487aed44e"). InnerVolumeSpecName "kube-api-access-97mqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:56.223589 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.223564    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-envoy" pod "93367dd2-e078-4c5f-9e84-e8e5832bb87f" (UID: "93367dd2-e078-4c5f-9e84-e8e5832bb87f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:56.323626 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.323597    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93367dd2-e078-4c5f-9e84-e8e5832bb87f-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.323741 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.323629    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97mqt\" (UniqueName: \"kubernetes.io/projected/9fafb996-e969-4c99-be6b-421487aed44e-kube-api-access-97mqt\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:56.510969 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.510888    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"]
Oct 30 19:21:56.511185 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:56.511109    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7" podUID="74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" containerName="istio-proxy" containerID="cri-o://5c71ceabb5b00a02480c2ff052829c36145a2cfc7267ec4b77b1e23ca8e5d882" gracePeriod=30
Oct 30 19:21:57.008719 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:57.008686    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"
Oct 30 19:21:57.009205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:57.008688    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm" event={"ID":"9fafb996-e969-4c99-be6b-421487aed44e","Type":"ContainerDied","Data":"a0a89af93469ec296d9a5018b64962b420bdb0c43ea7040219ae8f977e424d1b"}
Oct 30 19:21:57.009205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:57.008921    4372 scope.go:122] "RemoveContainer" containerID="64d8cfc1735e89a188a9a10309d67aafa966779130826b2e631db3df3581639d"
Oct 30 19:21:57.010469 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:57.010305    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"
Oct 30 19:21:57.010469 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:57.010347    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq" event={"ID":"93367dd2-e078-4c5f-9e84-e8e5832bb87f","Type":"ContainerDied","Data":"a9c35e3f9aae6639a06637917b233f75cd1774b6280698267169670ba47a29cd"}
Oct 30 19:21:57.019521 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:57.019503    4372 scope.go:122] "RemoveContainer" containerID="498e0226236c63e590deab53af930a68805a63adfe4d7057c31e91d4d5c4d9c9"
Oct 30 19:21:57.035584 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:57.035562    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"]
Oct 30 19:21:57.041992 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:57.041970    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-m74mq"]
Oct 30 19:21:57.055450 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:57.055430    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"]
Oct 30 19:21:57.060427 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:57.060406    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-nczzm"]
Oct 30 19:21:57.695594 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:21:57.695543    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:21:58.265926 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:58.265893    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93367dd2-e078-4c5f-9e84-e8e5832bb87f" path="/var/lib/kubelet/pods/93367dd2-e078-4c5f-9e84-e8e5832bb87f/volumes"
Oct 30 19:21:58.268538 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:58.266382    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fafb996-e969-4c99-be6b-421487aed44e" path="/var/lib/kubelet/pods/9fafb996-e969-4c99-be6b-421487aed44e/volumes"
Oct 30 19:21:58.802352 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:58.802323    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"]
Oct 30 19:21:58.808551 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:21:58.808522    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"]
Oct 30 19:22:00.812732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:00.812699    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:22:01.838359 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.838323    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:22:01.915147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.908630    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-credential-socket\") pod \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") "
Oct 30 19:22:01.915147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.908714    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-workload-certs\") pod \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") "
Oct 30 19:22:01.915147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.908850    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-envoy\") pod \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") "
Oct 30 19:22:01.915147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.908882    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istiod-ca-cert\") pod \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") "
Oct 30 19:22:01.915147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.908933    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-kube-api-access-nq6nj\" (UniqueName: \"kubernetes.io/projected/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-kube-api-access-nq6nj\") pod \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") "
Oct 30 19:22:01.915147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.908984    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-podinfo\") pod \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") "
Oct 30 19:22:01.915147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.909036    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-token\" (UniqueName: \"kubernetes.io/projected/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-token\") pod \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") "
Oct 30 19:22:01.915147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.909077    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-workload-socket\") pod \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") "
Oct 30 19:22:01.915147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.909220    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-data\") pod \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\" (UID: \"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb\") "
Oct 30 19:22:01.915147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.910249    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-data" pod "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" (UID: "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:01.915147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.910522    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-credential-socket" pod "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" (UID: "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:01.915147 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.910758    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-workload-certs" pod "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" (UID: "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:01.927065 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.921560    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-workload-socket" pod "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" (UID: "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:01.927065 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.921866    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istiod-ca-cert" pod "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" (UID: "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:01.952563 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.946815    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-kube-api-access-nq6nj" pod "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" (UID: "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb"). InnerVolumeSpecName "kube-api-access-nq6nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:01.971521 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.966001    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-podinfo" pod "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" (UID: "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:22:01.971521 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.966148    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-envoy" pod "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" (UID: "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:01.971521 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:01.971186    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-token" pod "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" (UID: "74ac9c50-e716-4e3e-9b75-befcdb4e4dbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:02.011183 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.011148    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.011183 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.011183    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.011466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.011198    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.011466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.011210    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.011466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.011224    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.011466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.011236    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq6nj\" (UniqueName: \"kubernetes.io/projected/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-kube-api-access-nq6nj\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.011466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.011248    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.011466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.011260    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.011466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.011273    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:02.034001 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.033960    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"]
Oct 30 19:22:02.037430 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.037400    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"]
Oct 30 19:22:02.055680 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.055643    4372 generic.go:356] "Generic (PLEG): container finished" podID="74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" containerID="5c71ceabb5b00a02480c2ff052829c36145a2cfc7267ec4b77b1e23ca8e5d882" exitCode=0
Oct 30 19:22:02.055855 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.055755    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7" event={"ID":"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb","Type":"ContainerDied","Data":"5c71ceabb5b00a02480c2ff052829c36145a2cfc7267ec4b77b1e23ca8e5d882"}
Oct 30 19:22:02.055855 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.055832    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7" event={"ID":"74ac9c50-e716-4e3e-9b75-befcdb4e4dbb","Type":"ContainerDied","Data":"cc64fb73e29610b5c34afcb4d782792833c10d47228033a2aaef6c7b4cccade4"}
Oct 30 19:22:02.055998 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.055860    4372 scope.go:122] "RemoveContainer" containerID="5c71ceabb5b00a02480c2ff052829c36145a2cfc7267ec4b77b1e23ca8e5d882"
Oct 30 19:22:02.056064 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.056019    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"
Oct 30 19:22:02.068494 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.068472    4372 scope.go:122] "RemoveContainer" containerID="5c71ceabb5b00a02480c2ff052829c36145a2cfc7267ec4b77b1e23ca8e5d882"
Oct 30 19:22:02.068784 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:02.068764    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c71ceabb5b00a02480c2ff052829c36145a2cfc7267ec4b77b1e23ca8e5d882\": container with ID starting with 5c71ceabb5b00a02480c2ff052829c36145a2cfc7267ec4b77b1e23ca8e5d882 not found: ID does not exist" containerID="5c71ceabb5b00a02480c2ff052829c36145a2cfc7267ec4b77b1e23ca8e5d882"
Oct 30 19:22:02.068867 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.068792    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c71ceabb5b00a02480c2ff052829c36145a2cfc7267ec4b77b1e23ca8e5d882"} err="failed to get container status \"5c71ceabb5b00a02480c2ff052829c36145a2cfc7267ec4b77b1e23ca8e5d882\": rpc error: code = NotFound desc = could not find container \"5c71ceabb5b00a02480c2ff052829c36145a2cfc7267ec4b77b1e23ca8e5d882\": container with ID starting with 5c71ceabb5b00a02480c2ff052829c36145a2cfc7267ec4b77b1e23ca8e5d882 not found: ID does not exist"
Oct 30 19:22:02.252350 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.252317    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"]
Oct 30 19:22:02.255298 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.255269    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d7870-openshift-default-7d84fc478f-t9ml7"]
Oct 30 19:22:02.264943 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.264918    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" path="/var/lib/kubelet/pods/74ac9c50-e716-4e3e-9b75-befcdb4e4dbb/volumes"
Oct 30 19:22:02.515969 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.515935    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg" podUID="925f7b63-9cb5-4ee8-8867-91530abaf46c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:22:02.516148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:02.515947    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj" podUID="31d23f77-b321-4ed8-b1e2-e0d1376d5697" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:22:03.803585 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:03.803546    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9" podUID="0e7d26de-2772-4819-88ed-4ab9dc24bba0" containerName="payload-processing" containerID="cri-o://f6f2db1f2205682e93a1fc6e0b33d0b652f71ff0a4d4e8f00801afcb7d31cb7e" gracePeriod=25
Oct 30 19:22:03.810025 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:03.809992    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6" podUID="008d989a-f0e1-4611-8206-0986d02786a3" containerName="payload-pre-processing" containerID="cri-o://e565fdb9a12c0275c3603db5a14ff2944684e3d266e2186cdb7b4e67c8248a4e" gracePeriod=25
Oct 30 19:22:05.579371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:05.579344    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"
Oct 30 19:22:05.619255 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:05.619227    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"
Oct 30 19:22:05.660901 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:05.660872    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d23f77-b321-4ed8-b1e2-e0d1376d5697-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d23f77-b321-4ed8-b1e2-e0d1376d5697-plugins-config-volume\") pod \"31d23f77-b321-4ed8-b1e2-e0d1376d5697\" (UID: \"31d23f77-b321-4ed8-b1e2-e0d1376d5697\") "
Oct 30 19:22:05.661110 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:05.661096    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d23f77-b321-4ed8-b1e2-e0d1376d5697-kube-api-access-wp22k\" (UniqueName: \"kubernetes.io/projected/31d23f77-b321-4ed8-b1e2-e0d1376d5697-kube-api-access-wp22k\") pod \"31d23f77-b321-4ed8-b1e2-e0d1376d5697\" (UID: \"31d23f77-b321-4ed8-b1e2-e0d1376d5697\") "
Oct 30 19:22:05.661276 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:05.661249    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d23f77-b321-4ed8-b1e2-e0d1376d5697-plugins-config-volume" pod "31d23f77-b321-4ed8-b1e2-e0d1376d5697" (UID: "31d23f77-b321-4ed8-b1e2-e0d1376d5697"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:05.661575 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:05.661552    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d23f77-b321-4ed8-b1e2-e0d1376d5697-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:05.663809 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:05.663782    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d23f77-b321-4ed8-b1e2-e0d1376d5697-kube-api-access-wp22k" pod "31d23f77-b321-4ed8-b1e2-e0d1376d5697" (UID: "31d23f77-b321-4ed8-b1e2-e0d1376d5697"). InnerVolumeSpecName "kube-api-access-wp22k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:05.761914 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:05.761878    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/925f7b63-9cb5-4ee8-8867-91530abaf46c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/925f7b63-9cb5-4ee8-8867-91530abaf46c-plugins-config-volume\") pod \"925f7b63-9cb5-4ee8-8867-91530abaf46c\" (UID: \"925f7b63-9cb5-4ee8-8867-91530abaf46c\") "
Oct 30 19:22:05.762094 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:05.762042    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925f7b63-9cb5-4ee8-8867-91530abaf46c-kube-api-access-t9c8g\" (UniqueName: \"kubernetes.io/projected/925f7b63-9cb5-4ee8-8867-91530abaf46c-kube-api-access-t9c8g\") pod \"925f7b63-9cb5-4ee8-8867-91530abaf46c\" (UID: \"925f7b63-9cb5-4ee8-8867-91530abaf46c\") "
Oct 30 19:22:05.762348 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:05.762321    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925f7b63-9cb5-4ee8-8867-91530abaf46c-plugins-config-volume" pod "925f7b63-9cb5-4ee8-8867-91530abaf46c" (UID: "925f7b63-9cb5-4ee8-8867-91530abaf46c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:05.762567 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:05.762550    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/925f7b63-9cb5-4ee8-8867-91530abaf46c-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:05.762637 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:05.762573    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp22k\" (UniqueName: \"kubernetes.io/projected/31d23f77-b321-4ed8-b1e2-e0d1376d5697-kube-api-access-wp22k\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:05.764236 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:05.764216    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925f7b63-9cb5-4ee8-8867-91530abaf46c-kube-api-access-t9c8g" pod "925f7b63-9cb5-4ee8-8867-91530abaf46c" (UID: "925f7b63-9cb5-4ee8-8867-91530abaf46c"). InnerVolumeSpecName "kube-api-access-t9c8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:05.863600 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:05.863567    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9c8g\" (UniqueName: \"kubernetes.io/projected/925f7b63-9cb5-4ee8-8867-91530abaf46c-kube-api-access-t9c8g\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:06.095287 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.095248    4372 generic.go:356] "Generic (PLEG): container finished" podID="925f7b63-9cb5-4ee8-8867-91530abaf46c" containerID="6a23df6fdc6c45da962a2f48b6ea1c831a4bc3eab426affc3766c7a98e26e256" exitCode=0
Oct 30 19:22:06.095287 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.095275    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"
Oct 30 19:22:06.095588 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.095412    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg" event={"ID":"925f7b63-9cb5-4ee8-8867-91530abaf46c","Type":"ContainerDied","Data":"6a23df6fdc6c45da962a2f48b6ea1c831a4bc3eab426affc3766c7a98e26e256"}
Oct 30 19:22:06.095588 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.095495    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg" event={"ID":"925f7b63-9cb5-4ee8-8867-91530abaf46c","Type":"ContainerDied","Data":"16c5a1ab4478ae742975b28dfd5ee702dbd68e314ee3c7ea5d7980e28f13a785"}
Oct 30 19:22:06.095588 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.095525    4372 scope.go:122] "RemoveContainer" containerID="6a23df6fdc6c45da962a2f48b6ea1c831a4bc3eab426affc3766c7a98e26e256"
Oct 30 19:22:06.106497 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.106471    4372 generic.go:356] "Generic (PLEG): container finished" podID="31d23f77-b321-4ed8-b1e2-e0d1376d5697" containerID="f39689a3d7ca0e8f1404c4dbddaf919c832160c01853e7f69cf93c0b8491bb08" exitCode=0
Oct 30 19:22:06.106622 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.106541    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"
Oct 30 19:22:06.106710 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.106543    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj" event={"ID":"31d23f77-b321-4ed8-b1e2-e0d1376d5697","Type":"ContainerDied","Data":"f39689a3d7ca0e8f1404c4dbddaf919c832160c01853e7f69cf93c0b8491bb08"}
Oct 30 19:22:06.106802 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.106750    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj" event={"ID":"31d23f77-b321-4ed8-b1e2-e0d1376d5697","Type":"ContainerDied","Data":"fb87e4773f09879e11f18385a187ad614ac4fca144a9e786e9ca4574d4631586"}
Oct 30 19:22:06.112354 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.112336    4372 scope.go:122] "RemoveContainer" containerID="6a23df6fdc6c45da962a2f48b6ea1c831a4bc3eab426affc3766c7a98e26e256"
Oct 30 19:22:06.112650 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:06.112624    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a23df6fdc6c45da962a2f48b6ea1c831a4bc3eab426affc3766c7a98e26e256\": container with ID starting with 6a23df6fdc6c45da962a2f48b6ea1c831a4bc3eab426affc3766c7a98e26e256 not found: ID does not exist" containerID="6a23df6fdc6c45da962a2f48b6ea1c831a4bc3eab426affc3766c7a98e26e256"
Oct 30 19:22:06.112732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.112662    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a23df6fdc6c45da962a2f48b6ea1c831a4bc3eab426affc3766c7a98e26e256"} err="failed to get container status \"6a23df6fdc6c45da962a2f48b6ea1c831a4bc3eab426affc3766c7a98e26e256\": rpc error: code = NotFound desc = could not find container \"6a23df6fdc6c45da962a2f48b6ea1c831a4bc3eab426affc3766c7a98e26e256\": container with ID starting with 6a23df6fdc6c45da962a2f48b6ea1c831a4bc3eab426affc3766c7a98e26e256 not found: ID does not exist"
Oct 30 19:22:06.112732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.112686    4372 scope.go:122] "RemoveContainer" containerID="f39689a3d7ca0e8f1404c4dbddaf919c832160c01853e7f69cf93c0b8491bb08"
Oct 30 19:22:06.121935 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.121919    4372 scope.go:122] "RemoveContainer" containerID="f39689a3d7ca0e8f1404c4dbddaf919c832160c01853e7f69cf93c0b8491bb08"
Oct 30 19:22:06.122235 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:06.122213    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39689a3d7ca0e8f1404c4dbddaf919c832160c01853e7f69cf93c0b8491bb08\": container with ID starting with f39689a3d7ca0e8f1404c4dbddaf919c832160c01853e7f69cf93c0b8491bb08 not found: ID does not exist" containerID="f39689a3d7ca0e8f1404c4dbddaf919c832160c01853e7f69cf93c0b8491bb08"
Oct 30 19:22:06.122298 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.122243    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39689a3d7ca0e8f1404c4dbddaf919c832160c01853e7f69cf93c0b8491bb08"} err="failed to get container status \"f39689a3d7ca0e8f1404c4dbddaf919c832160c01853e7f69cf93c0b8491bb08\": rpc error: code = NotFound desc = could not find container \"f39689a3d7ca0e8f1404c4dbddaf919c832160c01853e7f69cf93c0b8491bb08\": container with ID starting with f39689a3d7ca0e8f1404c4dbddaf919c832160c01853e7f69cf93c0b8491bb08 not found: ID does not exist"
Oct 30 19:22:06.128275 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.128253    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"]
Oct 30 19:22:06.135413 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.135392    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d7870-847d564dff-k85mg"]
Oct 30 19:22:06.153363 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.153345    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"]
Oct 30 19:22:06.161398 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.161376    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6d7870-5f97cd5646-7zdvj"]
Oct 30 19:22:06.269920 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.269882    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d23f77-b321-4ed8-b1e2-e0d1376d5697" path="/var/lib/kubelet/pods/31d23f77-b321-4ed8-b1e2-e0d1376d5697/volumes"
Oct 30 19:22:06.270541 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:06.270517    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="925f7b63-9cb5-4ee8-8867-91530abaf46c" path="/var/lib/kubelet/pods/925f7b63-9cb5-4ee8-8867-91530abaf46c/volumes"
Oct 30 19:22:07.497633 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:07.497600    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"]
Oct 30 19:22:07.948041 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:07.947999    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6" podUID="008d989a-f0e1-4611-8206-0986d02786a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:22:07.948041 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:07.948034    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9" podUID="0e7d26de-2772-4819-88ed-4ab9dc24bba0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:22:09.982716 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:09.981883    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"]
Oct 30 19:22:09.982716 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:09.982088    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv" podUID="37daf29d-e5e7-49d4-a763-4b948871431f" containerName="maas-api" containerID="cri-o://f572f3b60656b6f6bb8d9c71d7274ab88e50f6778969af83cf78123268a28013" gracePeriod=30
Oct 30 19:22:10.137018 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:10.136934    4372 generic.go:356] "Generic (PLEG): container finished" podID="37daf29d-e5e7-49d4-a763-4b948871431f" containerID="f572f3b60656b6f6bb8d9c71d7274ab88e50f6778969af83cf78123268a28013" exitCode=0
Oct 30 19:22:10.137226 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:10.137032    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv" event={"ID":"37daf29d-e5e7-49d4-a763-4b948871431f","Type":"ContainerDied","Data":"f572f3b60656b6f6bb8d9c71d7274ab88e50f6778969af83cf78123268a28013"}
Oct 30 19:22:10.357182 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:10.357153    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:22:10.422510 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:10.422404    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37daf29d-e5e7-49d4-a763-4b948871431f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37daf29d-e5e7-49d4-a763-4b948871431f-maas-api-tls\") pod \"37daf29d-e5e7-49d4-a763-4b948871431f\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") "
Oct 30 19:22:10.422510 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:10.422455    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37daf29d-e5e7-49d4-a763-4b948871431f-kube-api-access-lhbz9\" (UniqueName: \"kubernetes.io/projected/37daf29d-e5e7-49d4-a763-4b948871431f-kube-api-access-lhbz9\") pod \"37daf29d-e5e7-49d4-a763-4b948871431f\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") "
Oct 30 19:22:10.422780 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:10.422622    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37daf29d-e5e7-49d4-a763-4b948871431f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37daf29d-e5e7-49d4-a763-4b948871431f-openshift-service-ca\") pod \"37daf29d-e5e7-49d4-a763-4b948871431f\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") "
Oct 30 19:22:10.422780 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:10.422660    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37daf29d-e5e7-49d4-a763-4b948871431f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/37daf29d-e5e7-49d4-a763-4b948871431f-metrics-certs\") pod \"37daf29d-e5e7-49d4-a763-4b948871431f\" (UID: \"37daf29d-e5e7-49d4-a763-4b948871431f\") "
Oct 30 19:22:10.423345 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:10.423312    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37daf29d-e5e7-49d4-a763-4b948871431f-openshift-service-ca" pod "37daf29d-e5e7-49d4-a763-4b948871431f" (UID: "37daf29d-e5e7-49d4-a763-4b948871431f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:10.425826 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:10.425795    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37daf29d-e5e7-49d4-a763-4b948871431f-maas-api-tls" pod "37daf29d-e5e7-49d4-a763-4b948871431f" (UID: "37daf29d-e5e7-49d4-a763-4b948871431f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:10.426209 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:10.426178    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37daf29d-e5e7-49d4-a763-4b948871431f-kube-api-access-lhbz9" pod "37daf29d-e5e7-49d4-a763-4b948871431f" (UID: "37daf29d-e5e7-49d4-a763-4b948871431f"). InnerVolumeSpecName "kube-api-access-lhbz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:10.426684 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:10.426657    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37daf29d-e5e7-49d4-a763-4b948871431f-metrics-certs" pod "37daf29d-e5e7-49d4-a763-4b948871431f" (UID: "37daf29d-e5e7-49d4-a763-4b948871431f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:10.524733 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:10.524706    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37daf29d-e5e7-49d4-a763-4b948871431f-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:10.524733 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:10.524734    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhbz9\" (UniqueName: \"kubernetes.io/projected/37daf29d-e5e7-49d4-a763-4b948871431f-kube-api-access-lhbz9\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:10.524882 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:10.524745    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37daf29d-e5e7-49d4-a763-4b948871431f-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:10.524882 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:10.524753    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37daf29d-e5e7-49d4-a763-4b948871431f-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:11.144783 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:11.144737    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"
Oct 30 19:22:11.145253 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:11.144811    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv" event={"ID":"37daf29d-e5e7-49d4-a763-4b948871431f","Type":"ContainerDied","Data":"3f22a036a64134c3f40f625ec9dd647f7e3e7860eb256400d0e53150813f33bf"}
Oct 30 19:22:11.145253 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:11.144879    4372 scope.go:122] "RemoveContainer" containerID="f572f3b60656b6f6bb8d9c71d7274ab88e50f6778969af83cf78123268a28013"
Oct 30 19:22:11.171540 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:11.171507    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"]
Oct 30 19:22:11.178486 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:11.178461    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-21d896-5846dfdbf5-vjpwv"]
Oct 30 19:22:12.264222 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:12.264191    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37daf29d-e5e7-49d4-a763-4b948871431f" path="/var/lib/kubelet/pods/37daf29d-e5e7-49d4-a763-4b948871431f/volumes"
Oct 30 19:22:12.498080 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:12.498037    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7" podUID="9196fb90-c915-443e-bd49-109a77351d03" containerName="payload-pre-processing" containerID="cri-o://7396b75ef46870db798a0db09cb742d2c715fe3d1df5dc51c920f00cd2fc0e8d" gracePeriod=25
Oct 30 19:22:15.745713 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:15.745671    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7" podUID="9196fb90-c915-443e-bd49-109a77351d03" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:22:17.068914 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.068872    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main" containerID="cri-o://76652f2e1acec527bdfb5fb62e69a941efbe5e34e4aecda1c125c34d8f24d1ff" gracePeriod=45
Oct 30 19:22:17.071939 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.071907    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" containerID="cri-o://bc32abe46e8c9a378a22b70501b47407c913cd9a2d9ca41664c8feb3a0435a64" gracePeriod=45
Oct 30 19:22:17.078378 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.078350    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": dial tcp 10.133.0.70:8000: connect: connection refused"
Oct 30 19:22:17.331531 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.331505    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 19:22:17.365684 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.365655    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 19:22:17.505027 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.504986    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-dshm\") pod \"af1f5a15-28b6-497e-8c16-295736adf63a\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") "
Oct 30 19:22:17.505281 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.505096    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-model-cache\") pod \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") "
Oct 30 19:22:17.505281 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.505165    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-home\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-home\") pod \"af1f5a15-28b6-497e-8c16-295736adf63a\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") "
Oct 30 19:22:17.505281 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.505200    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-kube-api-access-zrww8\" (UniqueName: \"kubernetes.io/projected/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-kube-api-access-zrww8\") pod \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") "
Oct 30 19:22:17.505281 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.505249    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-tls-certs\") pod \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") "
Oct 30 19:22:17.505549 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.505301    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-tmp-dir\") pod \"af1f5a15-28b6-497e-8c16-295736adf63a\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") "
Oct 30 19:22:17.505549 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.505370    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af1f5a15-28b6-497e-8c16-295736adf63a-tls-certs\" (UniqueName: \"kubernetes.io/secret/af1f5a15-28b6-497e-8c16-295736adf63a-tls-certs\") pod \"af1f5a15-28b6-497e-8c16-295736adf63a\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") "
Oct 30 19:22:17.505549 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.505423    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1f5a15-28b6-497e-8c16-295736adf63a-kube-api-access-prh6n\" (UniqueName: \"kubernetes.io/projected/af1f5a15-28b6-497e-8c16-295736adf63a-kube-api-access-prh6n\") pod \"af1f5a15-28b6-497e-8c16-295736adf63a\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") "
Oct 30 19:22:17.506272 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.505473    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-home" pod "af1f5a15-28b6-497e-8c16-295736adf63a" (UID: "af1f5a15-28b6-497e-8c16-295736adf63a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.506662 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.505730    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-tmp-dir" pod "af1f5a15-28b6-497e-8c16-295736adf63a" (UID: "af1f5a15-28b6-497e-8c16-295736adf63a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.506781 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.505528    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-tmp-dir\") pod \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") "
Oct 30 19:22:17.506931 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.506915    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-model-cache\") pod \"af1f5a15-28b6-497e-8c16-295736adf63a\" (UID: \"af1f5a15-28b6-497e-8c16-295736adf63a\") "
Oct 30 19:22:17.507045 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.507030    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-dshm\") pod \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") "
Oct 30 19:22:17.507203 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.507187    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-home\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-home\") pod \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\" (UID: \"cf72f91a-0fe0-46ee-8747-ee5e36e355d7\") "
Oct 30 19:22:17.511294 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.510183    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-dshm" pod "af1f5a15-28b6-497e-8c16-295736adf63a" (UID: "af1f5a15-28b6-497e-8c16-295736adf63a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.511294 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.510706    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-model-cache" pod "af1f5a15-28b6-497e-8c16-295736adf63a" (UID: "af1f5a15-28b6-497e-8c16-295736adf63a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.511294 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.510816    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1f5a15-28b6-497e-8c16-295736adf63a-kube-api-access-prh6n" pod "af1f5a15-28b6-497e-8c16-295736adf63a" (UID: "af1f5a15-28b6-497e-8c16-295736adf63a"). InnerVolumeSpecName "kube-api-access-prh6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:17.513104 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.513063    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af1f5a15-28b6-497e-8c16-295736adf63a-tls-certs" pod "af1f5a15-28b6-497e-8c16-295736adf63a" (UID: "af1f5a15-28b6-497e-8c16-295736adf63a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:17.513780 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.513736    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-model-cache" pod "cf72f91a-0fe0-46ee-8747-ee5e36e355d7" (UID: "cf72f91a-0fe0-46ee-8747-ee5e36e355d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.513888 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.513792    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-tmp-dir" pod "cf72f91a-0fe0-46ee-8747-ee5e36e355d7" (UID: "cf72f91a-0fe0-46ee-8747-ee5e36e355d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.513957 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.513900    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.513957 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.513928    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.514075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.514020    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-home" pod "cf72f91a-0fe0-46ee-8747-ee5e36e355d7" (UID: "cf72f91a-0fe0-46ee-8747-ee5e36e355d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.515937 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.515911    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-dshm" pod "cf72f91a-0fe0-46ee-8747-ee5e36e355d7" (UID: "cf72f91a-0fe0-46ee-8747-ee5e36e355d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:17.516448 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.516418    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-tls-certs" pod "cf72f91a-0fe0-46ee-8747-ee5e36e355d7" (UID: "cf72f91a-0fe0-46ee-8747-ee5e36e355d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:17.516550 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.516530    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-kube-api-access-zrww8" pod "cf72f91a-0fe0-46ee-8747-ee5e36e355d7" (UID: "cf72f91a-0fe0-46ee-8747-ee5e36e355d7"). InnerVolumeSpecName "kube-api-access-zrww8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:17.615408 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.615377    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.615408 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.615408    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1f5a15-28b6-497e-8c16-295736adf63a-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.615587 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.615420    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prh6n\" (UniqueName: \"kubernetes.io/projected/af1f5a15-28b6-497e-8c16-295736adf63a-kube-api-access-prh6n\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.615587 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.615430    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.615587 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.615439    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.615587 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.615449    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.615587 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.615457    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.615587 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.615465    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1f5a15-28b6-497e-8c16-295736adf63a-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.615587 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.615472    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.615587 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.615480    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrww8\" (UniqueName: \"kubernetes.io/projected/cf72f91a-0fe0-46ee-8747-ee5e36e355d7-kube-api-access-zrww8\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:17.948891 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.948803    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9" podUID="0e7d26de-2772-4819-88ed-4ab9dc24bba0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:22:17.949066 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:17.948803    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6" podUID="008d989a-f0e1-4611-8206-0986d02786a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:22:18.191870 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.191832    4372 generic.go:356] "Generic (PLEG): container finished" podID="af1f5a15-28b6-497e-8c16-295736adf63a" containerID="76652f2e1acec527bdfb5fb62e69a941efbe5e34e4aecda1c125c34d8f24d1ff" exitCode=0
Oct 30 19:22:18.192344 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.191884    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"
Oct 30 19:22:18.192344 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.191917    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" event={"ID":"af1f5a15-28b6-497e-8c16-295736adf63a","Type":"ContainerDied","Data":"76652f2e1acec527bdfb5fb62e69a941efbe5e34e4aecda1c125c34d8f24d1ff"}
Oct 30 19:22:18.192344 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.191967    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p" event={"ID":"af1f5a15-28b6-497e-8c16-295736adf63a","Type":"ContainerDied","Data":"1c22720cf5af5e038420bd24d25e6653b90df7c21d9f7349ac4a663f24421a1e"}
Oct 30 19:22:18.192344 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.191985    4372 scope.go:122] "RemoveContainer" containerID="76652f2e1acec527bdfb5fb62e69a941efbe5e34e4aecda1c125c34d8f24d1ff"
Oct 30 19:22:18.193607 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.193583    4372 generic.go:356] "Generic (PLEG): container finished" podID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerID="bc32abe46e8c9a378a22b70501b47407c913cd9a2d9ca41664c8feb3a0435a64" exitCode=0
Oct 30 19:22:18.193742 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.193668    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" event={"ID":"cf72f91a-0fe0-46ee-8747-ee5e36e355d7","Type":"ContainerDied","Data":"bc32abe46e8c9a378a22b70501b47407c913cd9a2d9ca41664c8feb3a0435a64"}
Oct 30 19:22:18.193742 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.193671    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"
Oct 30 19:22:18.193742 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.193693    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8" event={"ID":"cf72f91a-0fe0-46ee-8747-ee5e36e355d7","Type":"ContainerDied","Data":"45cb257feb95a2abd8dac7c0b547548abacf8349125c7b32afb4437d5675595c"}
Oct 30 19:22:18.202974 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.202881    4372 scope.go:122] "RemoveContainer" containerID="76652f2e1acec527bdfb5fb62e69a941efbe5e34e4aecda1c125c34d8f24d1ff"
Oct 30 19:22:18.206525 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:18.206407    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76652f2e1acec527bdfb5fb62e69a941efbe5e34e4aecda1c125c34d8f24d1ff\": container with ID starting with 76652f2e1acec527bdfb5fb62e69a941efbe5e34e4aecda1c125c34d8f24d1ff not found: ID does not exist" containerID="76652f2e1acec527bdfb5fb62e69a941efbe5e34e4aecda1c125c34d8f24d1ff"
Oct 30 19:22:18.206525 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.206444    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76652f2e1acec527bdfb5fb62e69a941efbe5e34e4aecda1c125c34d8f24d1ff"} err="failed to get container status \"76652f2e1acec527bdfb5fb62e69a941efbe5e34e4aecda1c125c34d8f24d1ff\": rpc error: code = NotFound desc = could not find container \"76652f2e1acec527bdfb5fb62e69a941efbe5e34e4aecda1c125c34d8f24d1ff\": container with ID starting with 76652f2e1acec527bdfb5fb62e69a941efbe5e34e4aecda1c125c34d8f24d1ff not found: ID does not exist"
Oct 30 19:22:18.206525 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.206465    4372 scope.go:122] "RemoveContainer" containerID="bc32abe46e8c9a378a22b70501b47407c913cd9a2d9ca41664c8feb3a0435a64"
Oct 30 19:22:18.215244 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.215222    4372 scope.go:122] "RemoveContainer" containerID="bc32abe46e8c9a378a22b70501b47407c913cd9a2d9ca41664c8feb3a0435a64"
Oct 30 19:22:18.215507 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:18.215484    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc32abe46e8c9a378a22b70501b47407c913cd9a2d9ca41664c8feb3a0435a64\": container with ID starting with bc32abe46e8c9a378a22b70501b47407c913cd9a2d9ca41664c8feb3a0435a64 not found: ID does not exist" containerID="bc32abe46e8c9a378a22b70501b47407c913cd9a2d9ca41664c8feb3a0435a64"
Oct 30 19:22:18.215587 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.215518    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc32abe46e8c9a378a22b70501b47407c913cd9a2d9ca41664c8feb3a0435a64"} err="failed to get container status \"bc32abe46e8c9a378a22b70501b47407c913cd9a2d9ca41664c8feb3a0435a64\": rpc error: code = NotFound desc = could not find container \"bc32abe46e8c9a378a22b70501b47407c913cd9a2d9ca41664c8feb3a0435a64\": container with ID starting with bc32abe46e8c9a378a22b70501b47407c913cd9a2d9ca41664c8feb3a0435a64 not found: ID does not exist"
Oct 30 19:22:18.223710 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.223688    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"]
Oct 30 19:22:18.228506 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.228485    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/distinct-2-w0-84c32a-kserve-c564d5d69-ffl6p"]
Oct 30 19:22:18.248768 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.248734    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"]
Oct 30 19:22:18.251025 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.251007    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-84c32a/distinct-w0-84c32a-kserve-df9987ccd-sk6b8"]
Oct 30 19:22:18.265685 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.265660    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" path="/var/lib/kubelet/pods/af1f5a15-28b6-497e-8c16-295736adf63a/volumes"
Oct 30 19:22:18.266079 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:18.266066    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" path="/var/lib/kubelet/pods/cf72f91a-0fe0-46ee-8747-ee5e36e355d7/volumes"
Oct 30 19:22:19.252902 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:19.252869    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"]
Oct 30 19:22:19.253320 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:19.253078    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k" podUID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerName="istio-proxy" containerID="cri-o://088cb1a291043e1ae52f28d86bbe4e63b14a95d3de0615968256942148e13c19" gracePeriod=30
Oct 30 19:22:20.194714 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:20.194675    4372 patch_prober.go:29] interesting pod/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:22:20.194892 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:20.194730    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k" podUID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:22:20.307409 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:20.307372    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:22:24.085236 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.085194    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"
Oct 30 19:22:24.112699 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.112673    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"
Oct 30 19:22:24.189241 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.189215    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7d26de-2772-4819-88ed-4ab9dc24bba0-kube-api-access-d5txg\" (UniqueName: \"kubernetes.io/projected/0e7d26de-2772-4819-88ed-4ab9dc24bba0-kube-api-access-d5txg\") pod \"0e7d26de-2772-4819-88ed-4ab9dc24bba0\" (UID: \"0e7d26de-2772-4819-88ed-4ab9dc24bba0\") "
Oct 30 19:22:24.189387 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.189269    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e7d26de-2772-4819-88ed-4ab9dc24bba0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7d26de-2772-4819-88ed-4ab9dc24bba0-plugins-config-volume\") pod \"0e7d26de-2772-4819-88ed-4ab9dc24bba0\" (UID: \"0e7d26de-2772-4819-88ed-4ab9dc24bba0\") "
Oct 30 19:22:24.189669 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.189648    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7d26de-2772-4819-88ed-4ab9dc24bba0-plugins-config-volume" pod "0e7d26de-2772-4819-88ed-4ab9dc24bba0" (UID: "0e7d26de-2772-4819-88ed-4ab9dc24bba0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:24.191556 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.191527    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7d26de-2772-4819-88ed-4ab9dc24bba0-kube-api-access-d5txg" pod "0e7d26de-2772-4819-88ed-4ab9dc24bba0" (UID: "0e7d26de-2772-4819-88ed-4ab9dc24bba0"). InnerVolumeSpecName "kube-api-access-d5txg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:24.230407 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.230381    4372 generic.go:356] "Generic (PLEG): container finished" podID="008d989a-f0e1-4611-8206-0986d02786a3" containerID="e565fdb9a12c0275c3603db5a14ff2944684e3d266e2186cdb7b4e67c8248a4e" exitCode=0
Oct 30 19:22:24.230523 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.230425    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"
Oct 30 19:22:24.230523 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.230478    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6" event={"ID":"008d989a-f0e1-4611-8206-0986d02786a3","Type":"ContainerDied","Data":"e565fdb9a12c0275c3603db5a14ff2944684e3d266e2186cdb7b4e67c8248a4e"}
Oct 30 19:22:24.230658 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.230534    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6" event={"ID":"008d989a-f0e1-4611-8206-0986d02786a3","Type":"ContainerDied","Data":"8eaa907c489797f059f8e63d5a7cb89bf4c4636c9b92263b87ed5a0821c2450d"}
Oct 30 19:22:24.230658 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.230564    4372 scope.go:122] "RemoveContainer" containerID="e565fdb9a12c0275c3603db5a14ff2944684e3d266e2186cdb7b4e67c8248a4e"
Oct 30 19:22:24.231754 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.231734    4372 generic.go:356] "Generic (PLEG): container finished" podID="0e7d26de-2772-4819-88ed-4ab9dc24bba0" containerID="f6f2db1f2205682e93a1fc6e0b33d0b652f71ff0a4d4e8f00801afcb7d31cb7e" exitCode=0
Oct 30 19:22:24.231840 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.231773    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"
Oct 30 19:22:24.231840 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.231820    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9" event={"ID":"0e7d26de-2772-4819-88ed-4ab9dc24bba0","Type":"ContainerDied","Data":"f6f2db1f2205682e93a1fc6e0b33d0b652f71ff0a4d4e8f00801afcb7d31cb7e"}
Oct 30 19:22:24.231925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.231869    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9" event={"ID":"0e7d26de-2772-4819-88ed-4ab9dc24bba0","Type":"ContainerDied","Data":"d611958449e0fea77e14c7f3c603e4bcde17c14270199ded0fdd9b8b23f3e6a8"}
Oct 30 19:22:24.240899 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.240884    4372 scope.go:122] "RemoveContainer" containerID="e565fdb9a12c0275c3603db5a14ff2944684e3d266e2186cdb7b4e67c8248a4e"
Oct 30 19:22:24.241208 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:24.241189    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e565fdb9a12c0275c3603db5a14ff2944684e3d266e2186cdb7b4e67c8248a4e\": container with ID starting with e565fdb9a12c0275c3603db5a14ff2944684e3d266e2186cdb7b4e67c8248a4e not found: ID does not exist" containerID="e565fdb9a12c0275c3603db5a14ff2944684e3d266e2186cdb7b4e67c8248a4e"
Oct 30 19:22:24.241300 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.241219    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e565fdb9a12c0275c3603db5a14ff2944684e3d266e2186cdb7b4e67c8248a4e"} err="failed to get container status \"e565fdb9a12c0275c3603db5a14ff2944684e3d266e2186cdb7b4e67c8248a4e\": rpc error: code = NotFound desc = could not find container \"e565fdb9a12c0275c3603db5a14ff2944684e3d266e2186cdb7b4e67c8248a4e\": container with ID starting with e565fdb9a12c0275c3603db5a14ff2944684e3d266e2186cdb7b4e67c8248a4e not found: ID does not exist"
Oct 30 19:22:24.241300 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.241243    4372 scope.go:122] "RemoveContainer" containerID="f6f2db1f2205682e93a1fc6e0b33d0b652f71ff0a4d4e8f00801afcb7d31cb7e"
Oct 30 19:22:24.249499 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.249480    4372 scope.go:122] "RemoveContainer" containerID="f6f2db1f2205682e93a1fc6e0b33d0b652f71ff0a4d4e8f00801afcb7d31cb7e"
Oct 30 19:22:24.249755 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:24.249736    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f2db1f2205682e93a1fc6e0b33d0b652f71ff0a4d4e8f00801afcb7d31cb7e\": container with ID starting with f6f2db1f2205682e93a1fc6e0b33d0b652f71ff0a4d4e8f00801afcb7d31cb7e not found: ID does not exist" containerID="f6f2db1f2205682e93a1fc6e0b33d0b652f71ff0a4d4e8f00801afcb7d31cb7e"
Oct 30 19:22:24.249830 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.249765    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f2db1f2205682e93a1fc6e0b33d0b652f71ff0a4d4e8f00801afcb7d31cb7e"} err="failed to get container status \"f6f2db1f2205682e93a1fc6e0b33d0b652f71ff0a4d4e8f00801afcb7d31cb7e\": rpc error: code = NotFound desc = could not find container \"f6f2db1f2205682e93a1fc6e0b33d0b652f71ff0a4d4e8f00801afcb7d31cb7e\": container with ID starting with f6f2db1f2205682e93a1fc6e0b33d0b652f71ff0a4d4e8f00801afcb7d31cb7e not found: ID does not exist"
Oct 30 19:22:24.257484 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.257464    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"]
Oct 30 19:22:24.266610 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.266594    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-21d896-67fdb65584-9ggl9"]
Oct 30 19:22:24.290272 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.290252    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/008d989a-f0e1-4611-8206-0986d02786a3-kube-api-access-whct4\" (UniqueName: \"kubernetes.io/projected/008d989a-f0e1-4611-8206-0986d02786a3-kube-api-access-whct4\") pod \"008d989a-f0e1-4611-8206-0986d02786a3\" (UID: \"008d989a-f0e1-4611-8206-0986d02786a3\") "
Oct 30 19:22:24.290364 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.290349    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/008d989a-f0e1-4611-8206-0986d02786a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/008d989a-f0e1-4611-8206-0986d02786a3-plugins-config-volume\") pod \"008d989a-f0e1-4611-8206-0986d02786a3\" (UID: \"008d989a-f0e1-4611-8206-0986d02786a3\") "
Oct 30 19:22:24.290642 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.290628    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5txg\" (UniqueName: \"kubernetes.io/projected/0e7d26de-2772-4819-88ed-4ab9dc24bba0-kube-api-access-d5txg\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.290685 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.290644    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7d26de-2772-4819-88ed-4ab9dc24bba0-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.290718 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.290686    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/008d989a-f0e1-4611-8206-0986d02786a3-plugins-config-volume" pod "008d989a-f0e1-4611-8206-0986d02786a3" (UID: "008d989a-f0e1-4611-8206-0986d02786a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:24.292392 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.292375    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008d989a-f0e1-4611-8206-0986d02786a3-kube-api-access-whct4" pod "008d989a-f0e1-4611-8206-0986d02786a3" (UID: "008d989a-f0e1-4611-8206-0986d02786a3"). InnerVolumeSpecName "kube-api-access-whct4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:24.391664 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.391634    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whct4\" (UniqueName: \"kubernetes.io/projected/008d989a-f0e1-4611-8206-0986d02786a3-kube-api-access-whct4\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.391800 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.391667    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/008d989a-f0e1-4611-8206-0986d02786a3-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.516778 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.516753    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:22:24.559655 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.559626    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"]
Oct 30 19:22:24.568590 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.568552    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-21d896-85568bd84c-8p8p6"]
Oct 30 19:22:24.695909 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.695841    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-envoy\") pod \"8b6ca59e-1c18-468a-8ba1-23aca6351823\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") "
Oct 30 19:22:24.696067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.695925    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b6ca59e-1c18-468a-8ba1-23aca6351823-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6ca59e-1c18-468a-8ba1-23aca6351823-istiod-ca-cert\") pod \"8b6ca59e-1c18-468a-8ba1-23aca6351823\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") "
Oct 30 19:22:24.696067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.695943    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-data\") pod \"8b6ca59e-1c18-468a-8ba1-23aca6351823\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") "
Oct 30 19:22:24.696067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.695976    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-workload-socket\") pod \"8b6ca59e-1c18-468a-8ba1-23aca6351823\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") "
Oct 30 19:22:24.696067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.696016    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6ca59e-1c18-468a-8ba1-23aca6351823-kube-api-access-rwkcd\" (UniqueName: \"kubernetes.io/projected/8b6ca59e-1c18-468a-8ba1-23aca6351823-kube-api-access-rwkcd\") pod \"8b6ca59e-1c18-468a-8ba1-23aca6351823\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") "
Oct 30 19:22:24.696067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.696035    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-podinfo\") pod \"8b6ca59e-1c18-468a-8ba1-23aca6351823\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") "
Oct 30 19:22:24.696461 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.696158    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-credential-socket\") pod \"8b6ca59e-1c18-468a-8ba1-23aca6351823\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") "
Oct 30 19:22:24.696461 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.696198    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-workload-certs\") pod \"8b6ca59e-1c18-468a-8ba1-23aca6351823\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") "
Oct 30 19:22:24.696461 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.696227    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-token\" (UniqueName: \"kubernetes.io/projected/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-token\") pod \"8b6ca59e-1c18-468a-8ba1-23aca6351823\" (UID: \"8b6ca59e-1c18-468a-8ba1-23aca6351823\") "
Oct 30 19:22:24.696461 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.696352    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-workload-socket" pod "8b6ca59e-1c18-468a-8ba1-23aca6351823" (UID: "8b6ca59e-1c18-468a-8ba1-23aca6351823"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:24.696461 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.696367    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-data" pod "8b6ca59e-1c18-468a-8ba1-23aca6351823" (UID: "8b6ca59e-1c18-468a-8ba1-23aca6351823"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:24.696739 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.696460    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6ca59e-1c18-468a-8ba1-23aca6351823-istiod-ca-cert" pod "8b6ca59e-1c18-468a-8ba1-23aca6351823" (UID: "8b6ca59e-1c18-468a-8ba1-23aca6351823"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:24.696739 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.696625    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-credential-socket" pod "8b6ca59e-1c18-468a-8ba1-23aca6351823" (UID: "8b6ca59e-1c18-468a-8ba1-23aca6351823"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:24.696887 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.696856    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.696954 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.696912    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.696954 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.696930    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6ca59e-1c18-468a-8ba1-23aca6351823-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.696954 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.696945    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.697084 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.696869    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-workload-certs" pod "8b6ca59e-1c18-468a-8ba1-23aca6351823" (UID: "8b6ca59e-1c18-468a-8ba1-23aca6351823"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:24.698302 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.698280    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-envoy" pod "8b6ca59e-1c18-468a-8ba1-23aca6351823" (UID: "8b6ca59e-1c18-468a-8ba1-23aca6351823"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:24.698404 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.698391    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6ca59e-1c18-468a-8ba1-23aca6351823-kube-api-access-rwkcd" pod "8b6ca59e-1c18-468a-8ba1-23aca6351823" (UID: "8b6ca59e-1c18-468a-8ba1-23aca6351823"). InnerVolumeSpecName "kube-api-access-rwkcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:24.698951 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.698929    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-podinfo" pod "8b6ca59e-1c18-468a-8ba1-23aca6351823" (UID: "8b6ca59e-1c18-468a-8ba1-23aca6351823"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:22:24.699020 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.698945    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-token" pod "8b6ca59e-1c18-468a-8ba1-23aca6351823" (UID: "8b6ca59e-1c18-468a-8ba1-23aca6351823"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:24.798033 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.798003    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwkcd\" (UniqueName: \"kubernetes.io/projected/8b6ca59e-1c18-468a-8ba1-23aca6351823-kube-api-access-rwkcd\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.798033 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.798029    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.798207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.798039    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.798207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.798047    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:24.798207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:24.798056    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6ca59e-1c18-468a-8ba1-23aca6351823-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:25.241444 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:25.241412    4372 generic.go:356] "Generic (PLEG): container finished" podID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerID="088cb1a291043e1ae52f28d86bbe4e63b14a95d3de0615968256942148e13c19" exitCode=0
Oct 30 19:22:25.241899 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:25.241463    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"
Oct 30 19:22:25.241899 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:25.241492    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k" event={"ID":"8b6ca59e-1c18-468a-8ba1-23aca6351823","Type":"ContainerDied","Data":"088cb1a291043e1ae52f28d86bbe4e63b14a95d3de0615968256942148e13c19"}
Oct 30 19:22:25.241899 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:25.241546    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k" event={"ID":"8b6ca59e-1c18-468a-8ba1-23aca6351823","Type":"ContainerDied","Data":"d449e9f704848f0e697663fec40f49cd439f1053ee440911606cee6cd213bd23"}
Oct 30 19:22:25.241899 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:25.241575    4372 scope.go:122] "RemoveContainer" containerID="088cb1a291043e1ae52f28d86bbe4e63b14a95d3de0615968256942148e13c19"
Oct 30 19:22:25.251654 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:25.251632    4372 scope.go:122] "RemoveContainer" containerID="ca4ddc6ed25aa2e36a7bb17470f3deea14e1f50bde13517ccbfd8a2b096691ee"
Oct 30 19:22:25.260554 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:25.260538    4372 scope.go:122] "RemoveContainer" containerID="088cb1a291043e1ae52f28d86bbe4e63b14a95d3de0615968256942148e13c19"
Oct 30 19:22:25.260797 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:25.260773    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088cb1a291043e1ae52f28d86bbe4e63b14a95d3de0615968256942148e13c19\": container with ID starting with 088cb1a291043e1ae52f28d86bbe4e63b14a95d3de0615968256942148e13c19 not found: ID does not exist" containerID="088cb1a291043e1ae52f28d86bbe4e63b14a95d3de0615968256942148e13c19"
Oct 30 19:22:25.260892 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:25.260801    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088cb1a291043e1ae52f28d86bbe4e63b14a95d3de0615968256942148e13c19"} err="failed to get container status \"088cb1a291043e1ae52f28d86bbe4e63b14a95d3de0615968256942148e13c19\": rpc error: code = NotFound desc = could not find container \"088cb1a291043e1ae52f28d86bbe4e63b14a95d3de0615968256942148e13c19\": container with ID starting with 088cb1a291043e1ae52f28d86bbe4e63b14a95d3de0615968256942148e13c19 not found: ID does not exist"
Oct 30 19:22:25.260892 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:25.260819    4372 scope.go:122] "RemoveContainer" containerID="ca4ddc6ed25aa2e36a7bb17470f3deea14e1f50bde13517ccbfd8a2b096691ee"
Oct 30 19:22:25.261095 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:25.261079    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca4ddc6ed25aa2e36a7bb17470f3deea14e1f50bde13517ccbfd8a2b096691ee\": container with ID starting with ca4ddc6ed25aa2e36a7bb17470f3deea14e1f50bde13517ccbfd8a2b096691ee not found: ID does not exist" containerID="ca4ddc6ed25aa2e36a7bb17470f3deea14e1f50bde13517ccbfd8a2b096691ee"
Oct 30 19:22:25.261166 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:25.261102    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4ddc6ed25aa2e36a7bb17470f3deea14e1f50bde13517ccbfd8a2b096691ee"} err="failed to get container status \"ca4ddc6ed25aa2e36a7bb17470f3deea14e1f50bde13517ccbfd8a2b096691ee\": rpc error: code = NotFound desc = could not find container \"ca4ddc6ed25aa2e36a7bb17470f3deea14e1f50bde13517ccbfd8a2b096691ee\": container with ID starting with ca4ddc6ed25aa2e36a7bb17470f3deea14e1f50bde13517ccbfd8a2b096691ee not found: ID does not exist"
Oct 30 19:22:25.268560 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:25.268529    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"]
Oct 30 19:22:25.273140 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:25.272718    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-21d896-openshift-default-5f8df8d78f-59b9k"]
Oct 30 19:22:25.745853 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:25.745815    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7" podUID="9196fb90-c915-443e-bd49-109a77351d03" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:22:26.263700 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:26.263666    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="008d989a-f0e1-4611-8206-0986d02786a3" path="/var/lib/kubelet/pods/008d989a-f0e1-4611-8206-0986d02786a3/volumes"
Oct 30 19:22:26.264107 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:26.264065    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e7d26de-2772-4819-88ed-4ab9dc24bba0" path="/var/lib/kubelet/pods/0e7d26de-2772-4819-88ed-4ab9dc24bba0/volumes"
Oct 30 19:22:26.264506 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:26.264492    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b6ca59e-1c18-468a-8ba1-23aca6351823" path="/var/lib/kubelet/pods/8b6ca59e-1c18-468a-8ba1-23aca6351823/volumes"
Oct 30 19:22:27.696769 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:27.696732    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:22:29.634169 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:29.633730    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"]
Oct 30 19:22:29.634169 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:29.633929    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk" podUID="f84cfd48-726a-4579-afcf-ddbe01ea2fdb" containerName="maas-api" containerID="cri-o://a789959828b3c01b927b17729844c79b14207dc1e0de070705e53703c4a5c1f9" gracePeriod=30
Oct 30 19:22:29.978417 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:29.978388    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 19:22:30.080250 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.080208    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-kube-api-access-r7q4j\" (UniqueName: \"kubernetes.io/projected/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-kube-api-access-r7q4j\") pod \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") "
Oct 30 19:22:30.080441 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.080277    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-maas-api-tls\") pod \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") "
Oct 30 19:22:30.080441 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.080325    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-openshift-service-ca\") pod \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") "
Oct 30 19:22:30.080441 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.080427    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-metrics-certs\") pod \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\" (UID: \"f84cfd48-726a-4579-afcf-ddbe01ea2fdb\") "
Oct 30 19:22:30.080677 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.080651    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-openshift-service-ca" pod "f84cfd48-726a-4579-afcf-ddbe01ea2fdb" (UID: "f84cfd48-726a-4579-afcf-ddbe01ea2fdb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:30.080950 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.080932    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:30.082615 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.082582    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-maas-api-tls" pod "f84cfd48-726a-4579-afcf-ddbe01ea2fdb" (UID: "f84cfd48-726a-4579-afcf-ddbe01ea2fdb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:30.082730 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.082641    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-metrics-certs" pod "f84cfd48-726a-4579-afcf-ddbe01ea2fdb" (UID: "f84cfd48-726a-4579-afcf-ddbe01ea2fdb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:30.082820 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.082771    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-kube-api-access-r7q4j" pod "f84cfd48-726a-4579-afcf-ddbe01ea2fdb" (UID: "f84cfd48-726a-4579-afcf-ddbe01ea2fdb"). InnerVolumeSpecName "kube-api-access-r7q4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:30.186602 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.186561    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:30.186602 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.186602    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7q4j\" (UniqueName: \"kubernetes.io/projected/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-kube-api-access-r7q4j\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:30.186884 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.186617    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f84cfd48-726a-4579-afcf-ddbe01ea2fdb-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:30.271287 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.271201    4372 generic.go:356] "Generic (PLEG): container finished" podID="f84cfd48-726a-4579-afcf-ddbe01ea2fdb" containerID="a789959828b3c01b927b17729844c79b14207dc1e0de070705e53703c4a5c1f9" exitCode=0
Oct 30 19:22:30.271287 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.271245    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"
Oct 30 19:22:30.271539 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.271288    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk" event={"ID":"f84cfd48-726a-4579-afcf-ddbe01ea2fdb","Type":"ContainerDied","Data":"a789959828b3c01b927b17729844c79b14207dc1e0de070705e53703c4a5c1f9"}
Oct 30 19:22:30.271539 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.271343    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk" event={"ID":"f84cfd48-726a-4579-afcf-ddbe01ea2fdb","Type":"ContainerDied","Data":"04deb78a82c7ff4cb0fc344940f30495ef7bae1d528fb45f0d7ad9bfe01cbd4d"}
Oct 30 19:22:30.271539 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.271370    4372 scope.go:122] "RemoveContainer" containerID="a789959828b3c01b927b17729844c79b14207dc1e0de070705e53703c4a5c1f9"
Oct 30 19:22:30.281197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.281177    4372 scope.go:122] "RemoveContainer" containerID="a789959828b3c01b927b17729844c79b14207dc1e0de070705e53703c4a5c1f9"
Oct 30 19:22:30.281503 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:30.281484    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a789959828b3c01b927b17729844c79b14207dc1e0de070705e53703c4a5c1f9\": container with ID starting with a789959828b3c01b927b17729844c79b14207dc1e0de070705e53703c4a5c1f9 not found: ID does not exist" containerID="a789959828b3c01b927b17729844c79b14207dc1e0de070705e53703c4a5c1f9"
Oct 30 19:22:30.281562 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.281511    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a789959828b3c01b927b17729844c79b14207dc1e0de070705e53703c4a5c1f9"} err="failed to get container status \"a789959828b3c01b927b17729844c79b14207dc1e0de070705e53703c4a5c1f9\": rpc error: code = NotFound desc = could not find container \"a789959828b3c01b927b17729844c79b14207dc1e0de070705e53703c4a5c1f9\": container with ID starting with a789959828b3c01b927b17729844c79b14207dc1e0de070705e53703c4a5c1f9 not found: ID does not exist"
Oct 30 19:22:30.349706 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.349673    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"]
Oct 30 19:22:30.359825 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:30.359798    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk"]
Oct 30 19:22:32.264281 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:32.264247    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84cfd48-726a-4579-afcf-ddbe01ea2fdb" path="/var/lib/kubelet/pods/f84cfd48-726a-4579-afcf-ddbe01ea2fdb/volumes"
Oct 30 19:22:32.790270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:32.790245    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"
Oct 30 19:22:32.840502 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:32.840461    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-84c32a-7d76bc98cc-v5zsk" podUID="f84cfd48-726a-4579-afcf-ddbe01ea2fdb" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.51:8443/health\": context deadline exceeded"
Oct 30 19:22:32.922490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:32.922457    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9196fb90-c915-443e-bd49-109a77351d03-kube-api-access-8c4lv\" (UniqueName: \"kubernetes.io/projected/9196fb90-c915-443e-bd49-109a77351d03-kube-api-access-8c4lv\") pod \"9196fb90-c915-443e-bd49-109a77351d03\" (UID: \"9196fb90-c915-443e-bd49-109a77351d03\") "
Oct 30 19:22:32.922629 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:32.922498    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9196fb90-c915-443e-bd49-109a77351d03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9196fb90-c915-443e-bd49-109a77351d03-plugins-config-volume\") pod \"9196fb90-c915-443e-bd49-109a77351d03\" (UID: \"9196fb90-c915-443e-bd49-109a77351d03\") "
Oct 30 19:22:32.922927 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:32.922905    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9196fb90-c915-443e-bd49-109a77351d03-plugins-config-volume" pod "9196fb90-c915-443e-bd49-109a77351d03" (UID: "9196fb90-c915-443e-bd49-109a77351d03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:32.924805 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:32.924786    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9196fb90-c915-443e-bd49-109a77351d03-kube-api-access-8c4lv" pod "9196fb90-c915-443e-bd49-109a77351d03" (UID: "9196fb90-c915-443e-bd49-109a77351d03"). InnerVolumeSpecName "kube-api-access-8c4lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:33.023850 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:33.023794    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c4lv\" (UniqueName: \"kubernetes.io/projected/9196fb90-c915-443e-bd49-109a77351d03-kube-api-access-8c4lv\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:33.023850 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:33.023818    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9196fb90-c915-443e-bd49-109a77351d03-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:33.290667 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:33.290587    4372 generic.go:356] "Generic (PLEG): container finished" podID="9196fb90-c915-443e-bd49-109a77351d03" containerID="7396b75ef46870db798a0db09cb742d2c715fe3d1df5dc51c920f00cd2fc0e8d" exitCode=0
Oct 30 19:22:33.290667 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:33.290633    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"
Oct 30 19:22:33.290667 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:33.290634    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7" event={"ID":"9196fb90-c915-443e-bd49-109a77351d03","Type":"ContainerDied","Data":"7396b75ef46870db798a0db09cb742d2c715fe3d1df5dc51c920f00cd2fc0e8d"}
Oct 30 19:22:33.291228 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:33.290701    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7" event={"ID":"9196fb90-c915-443e-bd49-109a77351d03","Type":"ContainerDied","Data":"09215dbfa3c33478793ea87f2e4fc1eed359d22d1e2603f09b406ae02429585d"}
Oct 30 19:22:33.291228 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:33.290735    4372 scope.go:122] "RemoveContainer" containerID="7396b75ef46870db798a0db09cb742d2c715fe3d1df5dc51c920f00cd2fc0e8d"
Oct 30 19:22:33.300855 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:33.300840    4372 scope.go:122] "RemoveContainer" containerID="7396b75ef46870db798a0db09cb742d2c715fe3d1df5dc51c920f00cd2fc0e8d"
Oct 30 19:22:33.301114 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:33.301095    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7396b75ef46870db798a0db09cb742d2c715fe3d1df5dc51c920f00cd2fc0e8d\": container with ID starting with 7396b75ef46870db798a0db09cb742d2c715fe3d1df5dc51c920f00cd2fc0e8d not found: ID does not exist" containerID="7396b75ef46870db798a0db09cb742d2c715fe3d1df5dc51c920f00cd2fc0e8d"
Oct 30 19:22:33.301196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:33.301151    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7396b75ef46870db798a0db09cb742d2c715fe3d1df5dc51c920f00cd2fc0e8d"} err="failed to get container status \"7396b75ef46870db798a0db09cb742d2c715fe3d1df5dc51c920f00cd2fc0e8d\": rpc error: code = NotFound desc = could not find container \"7396b75ef46870db798a0db09cb742d2c715fe3d1df5dc51c920f00cd2fc0e8d\": container with ID starting with 7396b75ef46870db798a0db09cb742d2c715fe3d1df5dc51c920f00cd2fc0e8d not found: ID does not exist"
Oct 30 19:22:33.315667 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:33.315646    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"]
Oct 30 19:22:33.320083 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:33.320061    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-84c32a-5bdcff887f-9mjk7"]
Oct 30 19:22:34.266258 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:34.266221    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9196fb90-c915-443e-bd49-109a77351d03" path="/var/lib/kubelet/pods/9196fb90-c915-443e-bd49-109a77351d03/volumes"
Oct 30 19:22:36.904251 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904219    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"]
Oct 30 19:22:36.904690 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904676    4372 cpu_manager.go:414] "containerMap: removing container" podUID="31d23f77-b321-4ed8-b1e2-e0d1376d5697" containerName="payload-processing"
Oct 30 19:22:36.904732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904691    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d23f77-b321-4ed8-b1e2-e0d1376d5697" containerName="payload-processing"
Oct 30 19:22:36.904732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904702    4372 cpu_manager.go:414] "containerMap: removing container" podUID="37daf29d-e5e7-49d4-a763-4b948871431f" containerName="maas-api"
Oct 30 19:22:36.904732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904708    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37daf29d-e5e7-49d4-a763-4b948871431f" containerName="maas-api"
Oct 30 19:22:36.904732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904720    4372 cpu_manager.go:414] "containerMap: removing container" podUID="925f7b63-9cb5-4ee8-8867-91530abaf46c" containerName="payload-pre-processing"
Oct 30 19:22:36.904732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904725    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925f7b63-9cb5-4ee8-8867-91530abaf46c" containerName="payload-pre-processing"
Oct 30 19:22:36.904732 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904733    4372 cpu_manager.go:414] "containerMap: removing container" podUID="e86c2cb2-8ae4-4a42-b277-3779c70b995a" containerName="payload-processing"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904738    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e86c2cb2-8ae4-4a42-b277-3779c70b995a" containerName="payload-processing"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904746    4372 cpu_manager.go:414] "containerMap: removing container" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904750    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904758    4372 cpu_manager.go:414] "containerMap: removing container" podUID="88adeb15-521e-4bf1-85bd-6b549ea5c460" containerName="payload-processing"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904763    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88adeb15-521e-4bf1-85bd-6b549ea5c460" containerName="payload-processing"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904770    4372 cpu_manager.go:414] "containerMap: removing container" podUID="008d989a-f0e1-4611-8206-0986d02786a3" containerName="payload-pre-processing"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904775    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008d989a-f0e1-4611-8206-0986d02786a3" containerName="payload-pre-processing"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904782    4372 cpu_manager.go:414] "containerMap: removing container" podUID="93367dd2-e078-4c5f-9e84-e8e5832bb87f" containerName="istio-proxy"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904787    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93367dd2-e078-4c5f-9e84-e8e5832bb87f" containerName="istio-proxy"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904799    4372 cpu_manager.go:414] "containerMap: removing container" podUID="74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" containerName="istio-proxy"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904803    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" containerName="istio-proxy"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904810    4372 cpu_manager.go:414] "containerMap: removing container" podUID="adfcf9c2-e42d-443c-ae5c-d734dbd4bff7" containerName="maas-api"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904815    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adfcf9c2-e42d-443c-ae5c-d734dbd4bff7" containerName="maas-api"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904824    4372 cpu_manager.go:414] "containerMap: removing container" podUID="2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56" containerName="maas-api"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904828    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56" containerName="maas-api"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904834    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f84cfd48-726a-4579-afcf-ddbe01ea2fdb" containerName="maas-api"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904838    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84cfd48-726a-4579-afcf-ddbe01ea2fdb" containerName="maas-api"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904851    4372 cpu_manager.go:414] "containerMap: removing container" podUID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerName="istio-proxy"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904856    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerName="istio-proxy"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904863    4372 cpu_manager.go:414] "containerMap: removing container" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904867    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904873    4372 cpu_manager.go:414] "containerMap: removing container" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904877    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904886    4372 cpu_manager.go:414] "containerMap: removing container" podUID="0e7d26de-2772-4819-88ed-4ab9dc24bba0" containerName="payload-processing"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904890    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7d26de-2772-4819-88ed-4ab9dc24bba0" containerName="payload-processing"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904896    4372 cpu_manager.go:414] "containerMap: removing container" podUID="b8447ca7-004b-480f-8d70-c1896a5aa6ca" containerName="payload-processing"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904900    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8447ca7-004b-480f-8d70-c1896a5aa6ca" containerName="payload-processing"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904908    4372 cpu_manager.go:414] "containerMap: removing container" podUID="9196fb90-c915-443e-bd49-109a77351d03" containerName="payload-pre-processing"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904913    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9196fb90-c915-443e-bd49-109a77351d03" containerName="payload-pre-processing"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904920    4372 cpu_manager.go:414] "containerMap: removing container" podUID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerName="istio-proxy"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904924    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerName="istio-proxy"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904931    4372 cpu_manager.go:414] "containerMap: removing container" podUID="9fafb996-e969-4c99-be6b-421487aed44e" containerName="istio-proxy"
Oct 30 19:22:36.904944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904935    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fafb996-e969-4c99-be6b-421487aed44e" containerName="istio-proxy"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.904996    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ac9c50-e716-4e3e-9b75-befcdb4e4dbb" containerName="istio-proxy"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905003    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="adfcf9c2-e42d-443c-ae5c-d734dbd4bff7" containerName="maas-api"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905009    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e48e0f2-9f2b-4e19-9dcd-67e4a13e6b56" containerName="maas-api"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905017    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84cfd48-726a-4579-afcf-ddbe01ea2fdb" containerName="maas-api"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905024    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerName="istio-proxy"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905030    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a3b3a8-b37f-4c86-99fa-4e4494987be7" containerName="main"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905035    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1f5a15-28b6-497e-8c16-295736adf63a" containerName="main"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905042    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7d26de-2772-4819-88ed-4ab9dc24bba0" containerName="payload-processing"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905047    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8447ca7-004b-480f-8d70-c1896a5aa6ca" containerName="payload-processing"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905053    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="9196fb90-c915-443e-bd49-109a77351d03" containerName="payload-pre-processing"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905059    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6ca59e-1c18-468a-8ba1-23aca6351823" containerName="istio-proxy"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905064    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fafb996-e969-4c99-be6b-421487aed44e" containerName="istio-proxy"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905069    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d23f77-b321-4ed8-b1e2-e0d1376d5697" containerName="payload-processing"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905074    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="37daf29d-e5e7-49d4-a763-4b948871431f" containerName="maas-api"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905079    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="e86c2cb2-8ae4-4a42-b277-3779c70b995a" containerName="payload-processing"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905083    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="925f7b63-9cb5-4ee8-8867-91530abaf46c" containerName="payload-pre-processing"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905089    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf72f91a-0fe0-46ee-8747-ee5e36e355d7" containerName="main"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905094    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="88adeb15-521e-4bf1-85bd-6b549ea5c460" containerName="payload-processing"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905100    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="008d989a-f0e1-4611-8206-0986d02786a3" containerName="payload-pre-processing"
Oct 30 19:22:36.906044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.905105    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="93367dd2-e078-4c5f-9e84-e8e5832bb87f" containerName="istio-proxy"
Oct 30 19:22:36.908270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.908254    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:36.921705 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.921681    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"]
Oct 30 19:22:36.965212 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.965185    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:36.965333 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.965216    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcg74\" (UniqueName: \"kubernetes.io/projected/6ad84e49-97ec-4894-b3be-63f8ac8456f9-kube-api-access-wcg74\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:36.965333 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.965258    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:36.965413 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.965346    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:36.965413 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.965402    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:36.965480 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.965426    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:36.965550 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.965528    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:36.965618 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.965606    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:36.965655 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:36.965638    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.066860 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.066833    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.067035 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.066874    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.067035 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.066897    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.067035 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.066915    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcg74\" (UniqueName: \"kubernetes.io/projected/6ad84e49-97ec-4894-b3be-63f8ac8456f9-kube-api-access-wcg74\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.067035 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.066950    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.067035 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.066972    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.067035 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.066999    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.067035 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.067020    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.067515 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.067080    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.067515 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.067332    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.067515 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.067403    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.067694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.067657    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.067874 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.067754    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.067919 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.067846    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.069419 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.069397    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.069730 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.069711    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.080588 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.080564    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.081044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.081027    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcg74\" (UniqueName: \"kubernetes.io/projected/6ad84e49-97ec-4894-b3be-63f8ac8456f9-kube-api-access-wcg74\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.221792 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.221732    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:37.526877 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.526850    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"]
Oct 30 19:22:37.528363 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:22:37.528333    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad84e49_97ec_4894_b3be_63f8ac8456f9.slice/crio-134458a04aca4bff4171d6a1efb0a3eebe76d56b7760f770ea6dd9ff910f51fb WatchSource:0}: Error finding container 134458a04aca4bff4171d6a1efb0a3eebe76d56b7760f770ea6dd9ff910f51fb: Status 404 returned error can't find the container with id 134458a04aca4bff4171d6a1efb0a3eebe76d56b7760f770ea6dd9ff910f51fb
Oct 30 19:22:37.530815 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.530781    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:22:37.530941 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.530854    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:22:37.530941 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:37.530884    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:22:38.196159 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:38.194942    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:22:38.343597 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:38.343475    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w" event={"ID":"6ad84e49-97ec-4894-b3be-63f8ac8456f9","Type":"ContainerStarted","Data":"db27c144e53e4c58f3d10c4de801ea911854a78d36c55438d9182344ce0087ce"}
Oct 30 19:22:38.343597 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:38.343548    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w" event={"ID":"6ad84e49-97ec-4894-b3be-63f8ac8456f9","Type":"ContainerStarted","Data":"134458a04aca4bff4171d6a1efb0a3eebe76d56b7760f770ea6dd9ff910f51fb"}
Oct 30 19:22:38.377343 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:38.377280    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w" podStartSLOduration=2.377261214 podStartE2EDuration="2.377261214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:38.373705246 +0000 UTC m=+2972.242123576" watchObservedRunningTime="2026-10-30 19:22:38.377261214 +0000 UTC m=+2972.245679544"
Oct 30 19:22:39.129899 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.129225    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"]
Oct 30 19:22:39.133853 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.133810    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.148894 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.148734    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fe8ece-openshift-default-dockercfg-5b78v\""
Oct 30 19:22:39.153502 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.152204    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"]
Oct 30 19:22:39.212838 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.212805    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfffs\" (UniqueName: \"kubernetes.io/projected/450a9e2f-36a4-419e-a586-981cda482d97-kube-api-access-nfffs\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.213256 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.212856    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/450a9e2f-36a4-419e-a586-981cda482d97-istio-token\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.213256 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.212969    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/450a9e2f-36a4-419e-a586-981cda482d97-istio-podinfo\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.213256 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.213004    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-istio-envoy\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.213256 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.213039    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-workload-certs\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.213256 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.213113    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-workload-socket\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.213256 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.213172    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/450a9e2f-36a4-419e-a586-981cda482d97-istiod-ca-cert\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.213256 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.213194    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-credential-socket\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.213256 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.213231    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-istio-data\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.222266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.222241    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:39.223884 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.223859    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:22:39.224014 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.223914    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w" podUID="6ad84e49-97ec-4894-b3be-63f8ac8456f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 19:22:39.318659 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.315102    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/450a9e2f-36a4-419e-a586-981cda482d97-istiod-ca-cert\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.318659 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.315191    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-credential-socket\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.318659 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.315260    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-istio-data\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.318659 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.315306    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfffs\" (UniqueName: \"kubernetes.io/projected/450a9e2f-36a4-419e-a586-981cda482d97-kube-api-access-nfffs\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.318659 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.315378    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/450a9e2f-36a4-419e-a586-981cda482d97-istio-token\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.318659 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.315470    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/450a9e2f-36a4-419e-a586-981cda482d97-istio-podinfo\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.318659 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.315510    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-istio-envoy\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.318659 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.315546    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-workload-certs\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.318659 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.315637    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-workload-socket\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.319509 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.319453    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/450a9e2f-36a4-419e-a586-981cda482d97-istiod-ca-cert\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.319912 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.319790    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-credential-socket\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.321300 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.320089    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-istio-data\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.342640 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.329813    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-workload-certs\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.342640 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.336477    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-workload-socket\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.382194 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.352612    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/450a9e2f-36a4-419e-a586-981cda482d97-istio-podinfo\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.382433 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.374466    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-istio-envoy\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.402149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.393899    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/450a9e2f-36a4-419e-a586-981cda482d97-istio-token\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.402149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.399886    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfffs\" (UniqueName: \"kubernetes.io/projected/450a9e2f-36a4-419e-a586-981cda482d97-kube-api-access-nfffs\") pod \"e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") " pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.454345 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.453317    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:39.811256 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.811229    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"]
Oct 30 19:22:39.812047 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:22:39.812018    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450a9e2f_36a4_419e_a586_981cda482d97.slice/crio-e08d1f08c830412073ddb446c30348a3e1368071fef3db16807d3f0e38126e54 WatchSource:0}: Error finding container e08d1f08c830412073ddb446c30348a3e1368071fef3db16807d3f0e38126e54: Status 404 returned error can't find the container with id e08d1f08c830412073ddb446c30348a3e1368071fef3db16807d3f0e38126e54
Oct 30 19:22:39.814507 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.814474    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:22:39.814580 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.814540    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:22:39.814580 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:39.814569    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:22:40.223155 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:40.223084    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:22:40.223155 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:40.223151    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w" podUID="6ad84e49-97ec-4894-b3be-63f8ac8456f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 19:22:40.318801 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:40.318762    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"]
Oct 30 19:22:40.322792 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:40.322768    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"]
Oct 30 19:22:40.323232 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:40.323115    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj" podUID="d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" containerName="istio-proxy" containerID="cri-o://97703e073dc9a4c56d886cd133fb18219695dd6d5986b8ea2e88cae561410843" gracePeriod=30
Oct 30 19:22:40.529171 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:40.528967    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:22:40.529413 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:40.529374    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj" podUID="d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:22:40.531247 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:40.531210    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v" event={"ID":"450a9e2f-36a4-419e-a586-981cda482d97","Type":"ContainerStarted","Data":"a733c275f1c2935a125d0ebcd73118214a3d414c342f9db476ea702321d5e396"}
Oct 30 19:22:40.531664 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:40.531637    4372 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w" secret="" err="secret \"e2e-worker-w0-84c32a-openshift-default-dockercfg-k6dxc\" not found"
Oct 30 19:22:40.531859 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:40.531833    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v" event={"ID":"450a9e2f-36a4-419e-a586-981cda482d97","Type":"ContainerStarted","Data":"e08d1f08c830412073ddb446c30348a3e1368071fef3db16807d3f0e38126e54"}
Oct 30 19:22:40.561048 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:40.560998    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v" podStartSLOduration=1.560981561 podStartE2EDuration="1.560981561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:40.558880867 +0000 UTC m=+2974.427299198" watchObservedRunningTime="2026-10-30 19:22:40.560981561 +0000 UTC m=+2974.429399891"
Oct 30 19:22:41.223040 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:41.223003    4372 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:22:41.223271 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:41.223063    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w" podUID="6ad84e49-97ec-4894-b3be-63f8ac8456f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 19:22:41.454455 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:41.454414    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:41.459867 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:41.459842    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:41.536509 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:41.536419    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w" podUID="6ad84e49-97ec-4894-b3be-63f8ac8456f9" containerName="istio-proxy" containerID="cri-o://db27c144e53e4c58f3d10c4de801ea911854a78d36c55438d9182344ce0087ce" gracePeriod=30
Oct 30 19:22:41.537032 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:41.536692    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:41.537552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:41.537480    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:22:41.727782 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:41.727745    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"]
Oct 30 19:22:41.728053 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:41.728020    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq" podUID="df604beb-2e90-4717-a9e5-c6786dfe4473" containerName="istio-proxy" containerID="cri-o://87ebf638102aa4f870231bfb79747e09bfb6de6b44c4ca895f8a8a6eb6856ddf" gracePeriod=30
Oct 30 19:22:41.730144 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:41.730104    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"]
Oct 30 19:22:41.730371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:41.730294    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6" podUID="f4c4e204-f99c-46d4-bbb8-340438118869" containerName="istio-proxy" containerID="cri-o://ff09c5b035f95b9146b23e82e1f24179a07446569ed5b3b38968354e8346d366" gracePeriod=30
Oct 30 19:22:44.733609 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:44.733571    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"]
Oct 30 19:22:44.737744 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:44.737719    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"
Oct 30 19:22:44.748925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:44.748901    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fe8ece\""
Oct 30 19:22:44.763923 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:44.763898    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"]
Oct 30 19:22:44.775107 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:44.775084    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"]
Oct 30 19:22:44.779082 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:44.779051    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"
Oct 30 19:22:44.793311 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:44.793290    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fe8ece-dockercfg-tfbrf\""
Oct 30 19:22:44.801296 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:44.801272    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"]
Oct 30 19:22:44.905186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:44.905152    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211df401-6e2e-4db2-a0c9-4e0d24532573-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m\" (UID: \"211df401-6e2e-4db2-a0c9-4e0d24532573\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"
Oct 30 19:22:44.905539 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:44.905507    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33dfa4ab-12fb-4444-b419-9c24113ea33e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx\" (UID: \"33dfa4ab-12fb-4444-b419-9c24113ea33e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"
Oct 30 19:22:44.905756 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:44.905740    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qspm\" (UniqueName: \"kubernetes.io/projected/211df401-6e2e-4db2-a0c9-4e0d24532573-kube-api-access-8qspm\") pod \"payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m\" (UID: \"211df401-6e2e-4db2-a0c9-4e0d24532573\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"
Oct 30 19:22:44.906319 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:44.906289    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vlhs\" (UniqueName: \"kubernetes.io/projected/33dfa4ab-12fb-4444-b419-9c24113ea33e-kube-api-access-5vlhs\") pod \"payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx\" (UID: \"33dfa4ab-12fb-4444-b419-9c24113ea33e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"
Oct 30 19:22:45.007386 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.007292    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vlhs\" (UniqueName: \"kubernetes.io/projected/33dfa4ab-12fb-4444-b419-9c24113ea33e-kube-api-access-5vlhs\") pod \"payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx\" (UID: \"33dfa4ab-12fb-4444-b419-9c24113ea33e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"
Oct 30 19:22:45.007386 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.007377    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211df401-6e2e-4db2-a0c9-4e0d24532573-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m\" (UID: \"211df401-6e2e-4db2-a0c9-4e0d24532573\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"
Oct 30 19:22:45.007651 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.007411    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33dfa4ab-12fb-4444-b419-9c24113ea33e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx\" (UID: \"33dfa4ab-12fb-4444-b419-9c24113ea33e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"
Oct 30 19:22:45.007651 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.007473    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qspm\" (UniqueName: \"kubernetes.io/projected/211df401-6e2e-4db2-a0c9-4e0d24532573-kube-api-access-8qspm\") pod \"payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m\" (UID: \"211df401-6e2e-4db2-a0c9-4e0d24532573\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"
Oct 30 19:22:45.008222 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.008198    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211df401-6e2e-4db2-a0c9-4e0d24532573-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m\" (UID: \"211df401-6e2e-4db2-a0c9-4e0d24532573\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"
Oct 30 19:22:45.008338 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.008227    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33dfa4ab-12fb-4444-b419-9c24113ea33e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx\" (UID: \"33dfa4ab-12fb-4444-b419-9c24113ea33e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"
Oct 30 19:22:45.021838 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.021813    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qspm\" (UniqueName: \"kubernetes.io/projected/211df401-6e2e-4db2-a0c9-4e0d24532573-kube-api-access-8qspm\") pod \"payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m\" (UID: \"211df401-6e2e-4db2-a0c9-4e0d24532573\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"
Oct 30 19:22:45.022006 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.021984    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vlhs\" (UniqueName: \"kubernetes.io/projected/33dfa4ab-12fb-4444-b419-9c24113ea33e-kube-api-access-5vlhs\") pod \"payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx\" (UID: \"33dfa4ab-12fb-4444-b419-9c24113ea33e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"
Oct 30 19:22:45.052074 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.052056    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"
Oct 30 19:22:45.093524 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.093399    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"
Oct 30 19:22:45.194068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.194032    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:22:45.194214 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.194096    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6" podUID="f4c4e204-f99c-46d4-bbb8-340438118869" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:22:45.332603 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.332574    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"]
Oct 30 19:22:45.334046 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:22:45.334012    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211df401_6e2e_4db2_a0c9_4e0d24532573.slice/crio-c2f076b87114dac8491c2df5d30067242af57ca53361cefe26916734e15aa902 WatchSource:0}: Error finding container c2f076b87114dac8491c2df5d30067242af57ca53361cefe26916734e15aa902: Status 404 returned error can't find the container with id c2f076b87114dac8491c2df5d30067242af57ca53361cefe26916734e15aa902
Oct 30 19:22:45.402465 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.402440    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"]
Oct 30 19:22:45.404079 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:22:45.404054    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33dfa4ab_12fb_4444_b419_9c24113ea33e.slice/crio-16fd16d632cbc386fbe94ac1d90ebb3800627f819c06b979d32f523799821cbb WatchSource:0}: Error finding container 16fd16d632cbc386fbe94ac1d90ebb3800627f819c06b979d32f523799821cbb: Status 404 returned error can't find the container with id 16fd16d632cbc386fbe94ac1d90ebb3800627f819c06b979d32f523799821cbb
Oct 30 19:22:45.561172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.553009    4372 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:22:45.561172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.553072    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq" podUID="df604beb-2e90-4717-a9e5-c6786dfe4473" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:22:45.563534 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.563489    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx" event={"ID":"33dfa4ab-12fb-4444-b419-9c24113ea33e","Type":"ContainerStarted","Data":"bd024ffbf4adf61812be97fc923d6391779fef790151d3cd3ffe3d1925aecae8"}
Oct 30 19:22:45.563726 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.563564    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx" event={"ID":"33dfa4ab-12fb-4444-b419-9c24113ea33e","Type":"ContainerStarted","Data":"16fd16d632cbc386fbe94ac1d90ebb3800627f819c06b979d32f523799821cbb"}
Oct 30 19:22:45.563726 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.563637    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"
Oct 30 19:22:45.566624 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.566300    4372 generic.go:356] "Generic (PLEG): container finished" podID="d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" containerID="97703e073dc9a4c56d886cd133fb18219695dd6d5986b8ea2e88cae561410843" exitCode=0
Oct 30 19:22:45.566624 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.566336    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj" event={"ID":"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e","Type":"ContainerDied","Data":"97703e073dc9a4c56d886cd133fb18219695dd6d5986b8ea2e88cae561410843"}
Oct 30 19:22:45.568195 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.568172    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m" event={"ID":"211df401-6e2e-4db2-a0c9-4e0d24532573","Type":"ContainerStarted","Data":"56c752be625aa21e550494c7f32f4910c42705260b4a75fa1c45fb44bcbdacbe"}
Oct 30 19:22:45.568303 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.568218    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m" event={"ID":"211df401-6e2e-4db2-a0c9-4e0d24532573","Type":"ContainerStarted","Data":"c2f076b87114dac8491c2df5d30067242af57ca53361cefe26916734e15aa902"}
Oct 30 19:22:45.568303 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.568245    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"
Oct 30 19:22:45.579974 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.579948    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 19:22:45.590687 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.590604    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx" podStartSLOduration=1.590589399 podStartE2EDuration="1.590589399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:45.589401969 +0000 UTC m=+2979.457820299" watchObservedRunningTime="2026-10-30 19:22:45.590589399 +0000 UTC m=+2979.459007733"
Oct 30 19:22:45.642369 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.642309    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m" podStartSLOduration=1.64229447 podStartE2EDuration="1.64229447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:45.641216003 +0000 UTC m=+2979.509634337" watchObservedRunningTime="2026-10-30 19:22:45.64229447 +0000 UTC m=+2979.510712799"
Oct 30 19:22:45.715488 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.715459    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istiod-ca-cert\") pod \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") "
Oct 30 19:22:45.715641 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.715496    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-envoy\") pod \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") "
Oct 30 19:22:45.715641 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.715554    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-kube-api-access-kbdgh\" (UniqueName: \"kubernetes.io/projected/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-kube-api-access-kbdgh\") pod \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") "
Oct 30 19:22:45.715641 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.715601    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-workload-socket\") pod \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") "
Oct 30 19:22:45.715821 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.715649    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-workload-certs\") pod \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") "
Oct 30 19:22:45.715821 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.715740    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-credential-socket\") pod \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") "
Oct 30 19:22:45.715821 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.715781    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-token\" (UniqueName: \"kubernetes.io/projected/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-token\") pod \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") "
Oct 30 19:22:45.715986 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.715854    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-podinfo\") pod \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") "
Oct 30 19:22:45.715986 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.715887    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-data\") pod \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\" (UID: \"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e\") "
Oct 30 19:22:45.716104 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.715978    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-workload-socket" pod "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" (UID: "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:45.716104 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.716005    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istiod-ca-cert" pod "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" (UID: "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:45.716255 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.716179    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-credential-socket" pod "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" (UID: "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:45.716316 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.716262    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-workload-certs" pod "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" (UID: "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:45.716976 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.716622    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.716976 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.716649    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.716976 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.716665    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.716976 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.716680    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.716976 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.716934    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-data" pod "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" (UID: "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:45.718221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.718196    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-kube-api-access-kbdgh" pod "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" (UID: "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e"). InnerVolumeSpecName "kube-api-access-kbdgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:45.718549 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.718532    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-envoy" pod "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" (UID: "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:45.718617 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.718598    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-token" pod "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" (UID: "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:45.718884 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.718870    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-podinfo" pod "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" (UID: "d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:22:45.817671 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.817631    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.817671 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.817665    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbdgh\" (UniqueName: \"kubernetes.io/projected/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-kube-api-access-kbdgh\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.817671 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.817679    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.818196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.817693    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:45.818196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:45.817706    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:46.575120 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:46.575088    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"
Oct 30 19:22:46.575308 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:46.575141    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj" event={"ID":"d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e","Type":"ContainerDied","Data":"116b638b4b78a7d1625d57a0a6ac1446e087336b5d4cbfdac2cf7af95d6fc9b1"}
Oct 30 19:22:46.575308 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:46.575204    4372 scope.go:122] "RemoveContainer" containerID="97703e073dc9a4c56d886cd133fb18219695dd6d5986b8ea2e88cae561410843"
Oct 30 19:22:46.611995 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:46.611957    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"]
Oct 30 19:22:46.615772 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:46.615740    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-nc4sj"]
Oct 30 19:22:47.003469 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.003445    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:22:47.019833 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.019808    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:22:47.130015 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.129948    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-workload-socket\") pod \"df604beb-2e90-4717-a9e5-c6786dfe4473\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") "
Oct 30 19:22:47.130193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130017    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-data\") pod \"df604beb-2e90-4717-a9e5-c6786dfe4473\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") "
Oct 30 19:22:47.130193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130059    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df604beb-2e90-4717-a9e5-c6786dfe4473-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df604beb-2e90-4717-a9e5-c6786dfe4473-istiod-ca-cert\") pod \"df604beb-2e90-4717-a9e5-c6786dfe4473\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") "
Oct 30 19:22:47.130193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130106    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c4e204-f99c-46d4-bbb8-340438118869-kube-api-access-94shs\" (UniqueName: \"kubernetes.io/projected/f4c4e204-f99c-46d4-bbb8-340438118869-kube-api-access-94shs\") pod \"f4c4e204-f99c-46d4-bbb8-340438118869\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") "
Oct 30 19:22:47.130193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130158    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-workload-certs\") pod \"f4c4e204-f99c-46d4-bbb8-340438118869\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") "
Oct 30 19:22:47.130432 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130224    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-podinfo\") pod \"df604beb-2e90-4717-a9e5-c6786dfe4473\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") "
Oct 30 19:22:47.130432 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130277    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-envoy\") pod \"df604beb-2e90-4717-a9e5-c6786dfe4473\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") "
Oct 30 19:22:47.130432 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130294    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-data" pod "df604beb-2e90-4717-a9e5-c6786dfe4473" (UID: "df604beb-2e90-4717-a9e5-c6786dfe4473"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.130432 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130308    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-workload-socket\") pod \"f4c4e204-f99c-46d4-bbb8-340438118869\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") "
Oct 30 19:22:47.130432 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130341    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-credential-socket\") pod \"f4c4e204-f99c-46d4-bbb8-340438118869\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") "
Oct 30 19:22:47.130432 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130427    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4c4e204-f99c-46d4-bbb8-340438118869-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c4e204-f99c-46d4-bbb8-340438118869-istiod-ca-cert\") pod \"f4c4e204-f99c-46d4-bbb8-340438118869\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") "
Oct 30 19:22:47.130770 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130459    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4c4e204-f99c-46d4-bbb8-340438118869-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c4e204-f99c-46d4-bbb8-340438118869-istio-podinfo\") pod \"f4c4e204-f99c-46d4-bbb8-340438118869\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") "
Oct 30 19:22:47.130770 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130514    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c4e204-f99c-46d4-bbb8-340438118869-istio-token\" (UniqueName: \"kubernetes.io/projected/f4c4e204-f99c-46d4-bbb8-340438118869-istio-token\") pod \"f4c4e204-f99c-46d4-bbb8-340438118869\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") "
Oct 30 19:22:47.130770 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130543    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-workload-certs" pod "f4c4e204-f99c-46d4-bbb8-340438118869" (UID: "f4c4e204-f99c-46d4-bbb8-340438118869"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.130770 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130551    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df604beb-2e90-4717-a9e5-c6786dfe4473-istiod-ca-cert" pod "df604beb-2e90-4717-a9e5-c6786dfe4473" (UID: "df604beb-2e90-4717-a9e5-c6786dfe4473"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:47.130770 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130559    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df604beb-2e90-4717-a9e5-c6786dfe4473-kube-api-access-ppc66\" (UniqueName: \"kubernetes.io/projected/df604beb-2e90-4717-a9e5-c6786dfe4473-kube-api-access-ppc66\") pod \"df604beb-2e90-4717-a9e5-c6786dfe4473\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") "
Oct 30 19:22:47.130770 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130680    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-workload-socket" pod "f4c4e204-f99c-46d4-bbb8-340438118869" (UID: "f4c4e204-f99c-46d4-bbb8-340438118869"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.130770 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130735    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-istio-data\") pod \"f4c4e204-f99c-46d4-bbb8-340438118869\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") "
Oct 30 19:22:47.130770 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130772    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-credential-socket\") pod \"df604beb-2e90-4717-a9e5-c6786dfe4473\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") "
Oct 30 19:22:47.131250 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130810    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-workload-certs\") pod \"df604beb-2e90-4717-a9e5-c6786dfe4473\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") "
Oct 30 19:22:47.131250 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130859    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-istio-envoy\") pod \"f4c4e204-f99c-46d4-bbb8-340438118869\" (UID: \"f4c4e204-f99c-46d4-bbb8-340438118869\") "
Oct 30 19:22:47.131250 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130896    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-token\" (UniqueName: \"kubernetes.io/projected/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-token\") pod \"df604beb-2e90-4717-a9e5-c6786dfe4473\" (UID: \"df604beb-2e90-4717-a9e5-c6786dfe4473\") "
Oct 30 19:22:47.131250 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130898    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c4e204-f99c-46d4-bbb8-340438118869-istiod-ca-cert" pod "f4c4e204-f99c-46d4-bbb8-340438118869" (UID: "f4c4e204-f99c-46d4-bbb8-340438118869"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:47.131250 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.130309    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-workload-socket" pod "df604beb-2e90-4717-a9e5-c6786dfe4473" (UID: "df604beb-2e90-4717-a9e5-c6786dfe4473"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.131250 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.131205    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-credential-socket" pod "f4c4e204-f99c-46d4-bbb8-340438118869" (UID: "f4c4e204-f99c-46d4-bbb8-340438118869"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.131550 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.131490    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.131550 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.131511    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df604beb-2e90-4717-a9e5-c6786dfe4473-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.131550 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.131515    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-workload-certs" pod "df604beb-2e90-4717-a9e5-c6786dfe4473" (UID: "df604beb-2e90-4717-a9e5-c6786dfe4473"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.131550 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.131528    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.131550 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.131544    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.131826 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.131558    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.131826 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.131576    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c4e204-f99c-46d4-bbb8-340438118869-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.131826 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.131591    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.131826 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.131714    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-istio-data" pod "f4c4e204-f99c-46d4-bbb8-340438118869" (UID: "f4c4e204-f99c-46d4-bbb8-340438118869"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.131826 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.131782    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-credential-socket" pod "df604beb-2e90-4717-a9e5-c6786dfe4473" (UID: "df604beb-2e90-4717-a9e5-c6786dfe4473"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.133204 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.133176    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-envoy" pod "df604beb-2e90-4717-a9e5-c6786dfe4473" (UID: "df604beb-2e90-4717-a9e5-c6786dfe4473"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.133843 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.133815    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c4e204-f99c-46d4-bbb8-340438118869-kube-api-access-94shs" pod "f4c4e204-f99c-46d4-bbb8-340438118869" (UID: "f4c4e204-f99c-46d4-bbb8-340438118869"). InnerVolumeSpecName "kube-api-access-94shs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:47.134554 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.134219    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-istio-envoy" pod "f4c4e204-f99c-46d4-bbb8-340438118869" (UID: "f4c4e204-f99c-46d4-bbb8-340438118869"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:47.134879 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.134800    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c4e204-f99c-46d4-bbb8-340438118869-istio-token" pod "f4c4e204-f99c-46d4-bbb8-340438118869" (UID: "f4c4e204-f99c-46d4-bbb8-340438118869"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:47.134879 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.134838    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df604beb-2e90-4717-a9e5-c6786dfe4473-kube-api-access-ppc66" pod "df604beb-2e90-4717-a9e5-c6786dfe4473" (UID: "df604beb-2e90-4717-a9e5-c6786dfe4473"). InnerVolumeSpecName "kube-api-access-ppc66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:47.135066 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.135044    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-podinfo" pod "df604beb-2e90-4717-a9e5-c6786dfe4473" (UID: "df604beb-2e90-4717-a9e5-c6786dfe4473"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:22:47.135066 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.135050    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4c4e204-f99c-46d4-bbb8-340438118869-istio-podinfo" pod "f4c4e204-f99c-46d4-bbb8-340438118869" (UID: "f4c4e204-f99c-46d4-bbb8-340438118869"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:22:47.135231 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.135163    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-token" pod "df604beb-2e90-4717-a9e5-c6786dfe4473" (UID: "df604beb-2e90-4717-a9e5-c6786dfe4473"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:47.232934 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.232908    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94shs\" (UniqueName: \"kubernetes.io/projected/f4c4e204-f99c-46d4-bbb8-340438118869-kube-api-access-94shs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.232934 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.232933    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.233068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.232942    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.233068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.232955    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c4e204-f99c-46d4-bbb8-340438118869-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.233068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.232964    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c4e204-f99c-46d4-bbb8-340438118869-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.233068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.232973    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppc66\" (UniqueName: \"kubernetes.io/projected/df604beb-2e90-4717-a9e5-c6786dfe4473-kube-api-access-ppc66\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.233068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.232981    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.233068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.232990    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.233068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.232997    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df604beb-2e90-4717-a9e5-c6786dfe4473-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.233068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.233007    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c4e204-f99c-46d4-bbb8-340438118869-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.233068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.233015    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df604beb-2e90-4717-a9e5-c6786dfe4473-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:47.583116 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.583076    4372 generic.go:356] "Generic (PLEG): container finished" podID="f4c4e204-f99c-46d4-bbb8-340438118869" containerID="ff09c5b035f95b9146b23e82e1f24179a07446569ed5b3b38968354e8346d366" exitCode=0
Oct 30 19:22:47.583329 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.583153    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"
Oct 30 19:22:47.583329 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.583159    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6" event={"ID":"f4c4e204-f99c-46d4-bbb8-340438118869","Type":"ContainerDied","Data":"ff09c5b035f95b9146b23e82e1f24179a07446569ed5b3b38968354e8346d366"}
Oct 30 19:22:47.583329 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.583221    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6" event={"ID":"f4c4e204-f99c-46d4-bbb8-340438118869","Type":"ContainerDied","Data":"b5171aedf5072f2fc204fdfcb5300edb90d266f72ea8de535fe7d1b84629a174"}
Oct 30 19:22:47.583329 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.583251    4372 scope.go:122] "RemoveContainer" containerID="ff09c5b035f95b9146b23e82e1f24179a07446569ed5b3b38968354e8346d366"
Oct 30 19:22:47.585809 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.585787    4372 generic.go:356] "Generic (PLEG): container finished" podID="df604beb-2e90-4717-a9e5-c6786dfe4473" containerID="87ebf638102aa4f870231bfb79747e09bfb6de6b44c4ca895f8a8a6eb6856ddf" exitCode=0
Oct 30 19:22:47.585952 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.585839    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"
Oct 30 19:22:47.585952 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.585846    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq" event={"ID":"df604beb-2e90-4717-a9e5-c6786dfe4473","Type":"ContainerDied","Data":"87ebf638102aa4f870231bfb79747e09bfb6de6b44c4ca895f8a8a6eb6856ddf"}
Oct 30 19:22:47.585952 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.585876    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq" event={"ID":"df604beb-2e90-4717-a9e5-c6786dfe4473","Type":"ContainerDied","Data":"715a0f345fa5051391673f7bd3dda6c86388132e21fc2e1d05f9b5e15a152a5d"}
Oct 30 19:22:47.594591 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.594571    4372 scope.go:122] "RemoveContainer" containerID="ff09c5b035f95b9146b23e82e1f24179a07446569ed5b3b38968354e8346d366"
Oct 30 19:22:47.594953 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:47.594931    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff09c5b035f95b9146b23e82e1f24179a07446569ed5b3b38968354e8346d366\": container with ID starting with ff09c5b035f95b9146b23e82e1f24179a07446569ed5b3b38968354e8346d366 not found: ID does not exist" containerID="ff09c5b035f95b9146b23e82e1f24179a07446569ed5b3b38968354e8346d366"
Oct 30 19:22:47.595031 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.594961    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff09c5b035f95b9146b23e82e1f24179a07446569ed5b3b38968354e8346d366"} err="failed to get container status \"ff09c5b035f95b9146b23e82e1f24179a07446569ed5b3b38968354e8346d366\": rpc error: code = NotFound desc = could not find container \"ff09c5b035f95b9146b23e82e1f24179a07446569ed5b3b38968354e8346d366\": container with ID starting with ff09c5b035f95b9146b23e82e1f24179a07446569ed5b3b38968354e8346d366 not found: ID does not exist"
Oct 30 19:22:47.595031 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.594999    4372 scope.go:122] "RemoveContainer" containerID="87ebf638102aa4f870231bfb79747e09bfb6de6b44c4ca895f8a8a6eb6856ddf"
Oct 30 19:22:47.603828 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.603811    4372 scope.go:122] "RemoveContainer" containerID="87ebf638102aa4f870231bfb79747e09bfb6de6b44c4ca895f8a8a6eb6856ddf"
Oct 30 19:22:47.604071 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:47.604052    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ebf638102aa4f870231bfb79747e09bfb6de6b44c4ca895f8a8a6eb6856ddf\": container with ID starting with 87ebf638102aa4f870231bfb79747e09bfb6de6b44c4ca895f8a8a6eb6856ddf not found: ID does not exist" containerID="87ebf638102aa4f870231bfb79747e09bfb6de6b44c4ca895f8a8a6eb6856ddf"
Oct 30 19:22:47.604221 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.604079    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ebf638102aa4f870231bfb79747e09bfb6de6b44c4ca895f8a8a6eb6856ddf"} err="failed to get container status \"87ebf638102aa4f870231bfb79747e09bfb6de6b44c4ca895f8a8a6eb6856ddf\": rpc error: code = NotFound desc = could not find container \"87ebf638102aa4f870231bfb79747e09bfb6de6b44c4ca895f8a8a6eb6856ddf\": container with ID starting with 87ebf638102aa4f870231bfb79747e09bfb6de6b44c4ca895f8a8a6eb6856ddf not found: ID does not exist"
Oct 30 19:22:47.609555 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.609526    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"]
Oct 30 19:22:47.615044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.615020    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-t47x6"]
Oct 30 19:22:47.627857 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.627832    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"]
Oct 30 19:22:47.638240 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:47.638217    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-clxsq"]
Oct 30 19:22:48.267294 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:48.267262    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" path="/var/lib/kubelet/pods/d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e/volumes"
Oct 30 19:22:48.267710 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:48.267695    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df604beb-2e90-4717-a9e5-c6786dfe4473" path="/var/lib/kubelet/pods/df604beb-2e90-4717-a9e5-c6786dfe4473/volumes"
Oct 30 19:22:48.268105 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:48.268093    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4c4e204-f99c-46d4-bbb8-340438118869" path="/var/lib/kubelet/pods/f4c4e204-f99c-46d4-bbb8-340438118869/volumes"
Oct 30 19:22:49.601196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.601161    4372 generic.go:356] "Generic (PLEG): container finished" podID="6ad84e49-97ec-4894-b3be-63f8ac8456f9" containerID="db27c144e53e4c58f3d10c4de801ea911854a78d36c55438d9182344ce0087ce" exitCode=0
Oct 30 19:22:49.601557 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.601238    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w" event={"ID":"6ad84e49-97ec-4894-b3be-63f8ac8456f9","Type":"ContainerDied","Data":"db27c144e53e4c58f3d10c4de801ea911854a78d36c55438d9182344ce0087ce"}
Oct 30 19:22:49.701294 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.701268    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:49.757532 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.757411    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-envoy\") pod \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") "
Oct 30 19:22:49.757532 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.757530    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-data\") pod \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") "
Oct 30 19:22:49.757707 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.757588    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-workload-socket\") pod \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") "
Oct 30 19:22:49.757707 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.757626    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-token\" (UniqueName: \"kubernetes.io/projected/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-token\") pod \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") "
Oct 30 19:22:49.757707 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.757657    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-credential-socket\") pod \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") "
Oct 30 19:22:49.757925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.757708    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-workload-certs\") pod \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") "
Oct 30 19:22:49.757925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.757805    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad84e49-97ec-4894-b3be-63f8ac8456f9-kube-api-access-wcg74\" (UniqueName: \"kubernetes.io/projected/6ad84e49-97ec-4894-b3be-63f8ac8456f9-kube-api-access-wcg74\") pod \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") "
Oct 30 19:22:49.757925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.757866    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istiod-ca-cert\") pod \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") "
Oct 30 19:22:49.757925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.757855    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-data" pod "6ad84e49-97ec-4894-b3be-63f8ac8456f9" (UID: "6ad84e49-97ec-4894-b3be-63f8ac8456f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:49.757925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.757917    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-podinfo\") pod \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\" (UID: \"6ad84e49-97ec-4894-b3be-63f8ac8456f9\") "
Oct 30 19:22:49.758236 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.757931    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-workload-socket" pod "6ad84e49-97ec-4894-b3be-63f8ac8456f9" (UID: "6ad84e49-97ec-4894-b3be-63f8ac8456f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:49.758236 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.758009    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-credential-socket" pod "6ad84e49-97ec-4894-b3be-63f8ac8456f9" (UID: "6ad84e49-97ec-4894-b3be-63f8ac8456f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:49.758344 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.758243    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-workload-certs" pod "6ad84e49-97ec-4894-b3be-63f8ac8456f9" (UID: "6ad84e49-97ec-4894-b3be-63f8ac8456f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:49.758344 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.758300    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istiod-ca-cert" pod "6ad84e49-97ec-4894-b3be-63f8ac8456f9" (UID: "6ad84e49-97ec-4894-b3be-63f8ac8456f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:49.758629 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.758381    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:49.758629 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.758405    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:49.758629 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.758419    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:49.758629 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.758433    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:49.760422 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.760336    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-token" pod "6ad84e49-97ec-4894-b3be-63f8ac8456f9" (UID: "6ad84e49-97ec-4894-b3be-63f8ac8456f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:49.760527 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.760478    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-envoy" pod "6ad84e49-97ec-4894-b3be-63f8ac8456f9" (UID: "6ad84e49-97ec-4894-b3be-63f8ac8456f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:49.761000 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.760946    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad84e49-97ec-4894-b3be-63f8ac8456f9-kube-api-access-wcg74" pod "6ad84e49-97ec-4894-b3be-63f8ac8456f9" (UID: "6ad84e49-97ec-4894-b3be-63f8ac8456f9"). InnerVolumeSpecName "kube-api-access-wcg74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:49.761000 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.760985    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-podinfo" pod "6ad84e49-97ec-4894-b3be-63f8ac8456f9" (UID: "6ad84e49-97ec-4894-b3be-63f8ac8456f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:22:49.859282 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.859252    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcg74\" (UniqueName: \"kubernetes.io/projected/6ad84e49-97ec-4894-b3be-63f8ac8456f9-kube-api-access-wcg74\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:49.859282 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.859278    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:49.859435 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.859289    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:49.859435 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.859298    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:49.859435 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:49.859307    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad84e49-97ec-4894-b3be-63f8ac8456f9-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:50.608795 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:50.608767    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"
Oct 30 19:22:50.609300 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:50.608795    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w" event={"ID":"6ad84e49-97ec-4894-b3be-63f8ac8456f9","Type":"ContainerDied","Data":"134458a04aca4bff4171d6a1efb0a3eebe76d56b7760f770ea6dd9ff910f51fb"}
Oct 30 19:22:50.609300 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:50.608861    4372 scope.go:122] "RemoveContainer" containerID="db27c144e53e4c58f3d10c4de801ea911854a78d36c55438d9182344ce0087ce"
Oct 30 19:22:50.641115 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:50.641090    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"]
Oct 30 19:22:50.646492 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:50.646468    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-28l5w"]
Oct 30 19:22:51.235251 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.235215    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"]
Oct 30 19:22:51.235686 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.235672    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" containerName="istio-proxy"
Oct 30 19:22:51.235757 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.235688    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" containerName="istio-proxy"
Oct 30 19:22:51.235757 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.235708    4372 cpu_manager.go:414] "containerMap: removing container" podUID="df604beb-2e90-4717-a9e5-c6786dfe4473" containerName="istio-proxy"
Oct 30 19:22:51.235757 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.235714    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df604beb-2e90-4717-a9e5-c6786dfe4473" containerName="istio-proxy"
Oct 30 19:22:51.235757 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.235724    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f4c4e204-f99c-46d4-bbb8-340438118869" containerName="istio-proxy"
Oct 30 19:22:51.235757 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.235729    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4c4e204-f99c-46d4-bbb8-340438118869" containerName="istio-proxy"
Oct 30 19:22:51.235757 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.235741    4372 cpu_manager.go:414] "containerMap: removing container" podUID="6ad84e49-97ec-4894-b3be-63f8ac8456f9" containerName="istio-proxy"
Oct 30 19:22:51.235757 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.235745    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad84e49-97ec-4894-b3be-63f8ac8456f9" containerName="istio-proxy"
Oct 30 19:22:51.236012 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.235831    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a2d4ed-6bc5-417b-bef6-836ed18e9e7e" containerName="istio-proxy"
Oct 30 19:22:51.236012 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.235842    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="df604beb-2e90-4717-a9e5-c6786dfe4473" containerName="istio-proxy"
Oct 30 19:22:51.236012 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.235849    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4c4e204-f99c-46d4-bbb8-340438118869" containerName="istio-proxy"
Oct 30 19:22:51.236012 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.235857    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad84e49-97ec-4894-b3be-63f8ac8456f9" containerName="istio-proxy"
Oct 30 19:22:51.240627 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.240607    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:51.245490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.245470    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fe8ece-serving-cert\""
Oct 30 19:22:51.258168 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.258106    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"]
Oct 30 19:22:51.271097 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.271067    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f86acaf-a6dd-4962-82db-f2103f9f263f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:51.271275 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.271251    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f86acaf-a6dd-4962-82db-f2103f9f263f-maas-api-tls\") pod \"maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:51.271376 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.271353    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsxkt\" (UniqueName: \"kubernetes.io/projected/0f86acaf-a6dd-4962-82db-f2103f9f263f-kube-api-access-gsxkt\") pod \"maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:51.271519 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.271496    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f86acaf-a6dd-4962-82db-f2103f9f263f-metrics-certs\") pod \"maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:51.372791 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.372763    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f86acaf-a6dd-4962-82db-f2103f9f263f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:51.372927 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.372822    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f86acaf-a6dd-4962-82db-f2103f9f263f-maas-api-tls\") pod \"maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:51.372927 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.372855    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsxkt\" (UniqueName: \"kubernetes.io/projected/0f86acaf-a6dd-4962-82db-f2103f9f263f-kube-api-access-gsxkt\") pod \"maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:51.372927 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.372900    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f86acaf-a6dd-4962-82db-f2103f9f263f-metrics-certs\") pod \"maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:51.373430 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.373409    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f86acaf-a6dd-4962-82db-f2103f9f263f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:51.375567 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.375540    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f86acaf-a6dd-4962-82db-f2103f9f263f-maas-api-tls\") pod \"maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:51.375660 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.375548    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f86acaf-a6dd-4962-82db-f2103f9f263f-metrics-certs\") pod \"maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:51.392165 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.392144    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsxkt\" (UniqueName: \"kubernetes.io/projected/0f86acaf-a6dd-4962-82db-f2103f9f263f-kube-api-access-gsxkt\") pod \"maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:51.558389 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.558311    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:51.839706 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.839679    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"]
Oct 30 19:22:51.840661 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:22:51.840635    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f86acaf_a6dd_4962_82db_f2103f9f263f.slice/crio-921a7a4d970cdebb9b4aa7c63b1ab4995fa1ab3fd7920292ac9d7d0f55b97b89 WatchSource:0}: Error finding container 921a7a4d970cdebb9b4aa7c63b1ab4995fa1ab3fd7920292ac9d7d0f55b97b89: Status 404 returned error can't find the container with id 921a7a4d970cdebb9b4aa7c63b1ab4995fa1ab3fd7920292ac9d7d0f55b97b89
Oct 30 19:22:51.842108 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:51.842091    4372 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:22:52.265846 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:52.265756    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad84e49-97ec-4894-b3be-63f8ac8456f9" path="/var/lib/kubelet/pods/6ad84e49-97ec-4894-b3be-63f8ac8456f9/volumes"
Oct 30 19:22:52.421094 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:52.421068    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"]
Oct 30 19:22:52.440846 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:52.440815    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"]
Oct 30 19:22:52.625331 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:52.625298    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn" event={"ID":"0f86acaf-a6dd-4962-82db-f2103f9f263f","Type":"ContainerStarted","Data":"d0234e91e1b30dc5d6e0ae119bf141a1c0b745218d39aaf1c00f0b1e98032031"}
Oct 30 19:22:52.625498 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:52.625357    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn" event={"ID":"0f86acaf-a6dd-4962-82db-f2103f9f263f","Type":"ContainerStarted","Data":"921a7a4d970cdebb9b4aa7c63b1ab4995fa1ab3fd7920292ac9d7d0f55b97b89"}
Oct 30 19:22:52.625552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:52.625535    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:52.656282 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:52.656237    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn" podStartSLOduration=1.085732416 podStartE2EDuration="1.656223329s" totalImagesPullingTime="570.490913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:52.653339355 +0000 UTC m=+2986.521757722" watchObservedRunningTime="2026-10-30 19:22:52.656223329 +0000 UTC m=+2986.524641653"
Oct 30 19:22:54.119506 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.119469    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"]
Oct 30 19:22:54.119984 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.119802    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l" podUID="b4875864-541b-423c-9386-3535205fe52a" containerName="maas-api" containerID="cri-o://04123e157577e9704e44bfb6f63b69b5f31796ce3c662d0120ff08969dc69b53" gracePeriod=30
Oct 30 19:22:54.437283 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.437258    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 19:22:54.506378 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.506346    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4875864-541b-423c-9386-3535205fe52a-kube-api-access-rg7bq\" (UniqueName: \"kubernetes.io/projected/b4875864-541b-423c-9386-3535205fe52a-kube-api-access-rg7bq\") pod \"b4875864-541b-423c-9386-3535205fe52a\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") "
Oct 30 19:22:54.506567 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.506409    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4875864-541b-423c-9386-3535205fe52a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4875864-541b-423c-9386-3535205fe52a-openshift-service-ca\") pod \"b4875864-541b-423c-9386-3535205fe52a\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") "
Oct 30 19:22:54.506567 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.506453    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4875864-541b-423c-9386-3535205fe52a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4875864-541b-423c-9386-3535205fe52a-maas-api-tls\") pod \"b4875864-541b-423c-9386-3535205fe52a\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") "
Oct 30 19:22:54.506567 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.506473    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4875864-541b-423c-9386-3535205fe52a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4875864-541b-423c-9386-3535205fe52a-metrics-certs\") pod \"b4875864-541b-423c-9386-3535205fe52a\" (UID: \"b4875864-541b-423c-9386-3535205fe52a\") "
Oct 30 19:22:54.506875 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.506841    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4875864-541b-423c-9386-3535205fe52a-openshift-service-ca" pod "b4875864-541b-423c-9386-3535205fe52a" (UID: "b4875864-541b-423c-9386-3535205fe52a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:54.509330 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.509300    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4875864-541b-423c-9386-3535205fe52a-kube-api-access-rg7bq" pod "b4875864-541b-423c-9386-3535205fe52a" (UID: "b4875864-541b-423c-9386-3535205fe52a"). InnerVolumeSpecName "kube-api-access-rg7bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:54.509330 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.509317    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4875864-541b-423c-9386-3535205fe52a-metrics-certs" pod "b4875864-541b-423c-9386-3535205fe52a" (UID: "b4875864-541b-423c-9386-3535205fe52a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:54.510712 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.510689    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4875864-541b-423c-9386-3535205fe52a-maas-api-tls" pod "b4875864-541b-423c-9386-3535205fe52a" (UID: "b4875864-541b-423c-9386-3535205fe52a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:54.608193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.608166    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4875864-541b-423c-9386-3535205fe52a-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:54.608193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.608192    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4875864-541b-423c-9386-3535205fe52a-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:54.608406 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.608207    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg7bq\" (UniqueName: \"kubernetes.io/projected/b4875864-541b-423c-9386-3535205fe52a-kube-api-access-rg7bq\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:54.608406 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.608223    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4875864-541b-423c-9386-3535205fe52a-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:54.664782 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.664675    4372 generic.go:356] "Generic (PLEG): container finished" podID="b4875864-541b-423c-9386-3535205fe52a" containerID="04123e157577e9704e44bfb6f63b69b5f31796ce3c662d0120ff08969dc69b53" exitCode=0
Oct 30 19:22:54.664782 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.664726    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l" event={"ID":"b4875864-541b-423c-9386-3535205fe52a","Type":"ContainerDied","Data":"04123e157577e9704e44bfb6f63b69b5f31796ce3c662d0120ff08969dc69b53"}
Oct 30 19:22:54.664782 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.664772    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l" event={"ID":"b4875864-541b-423c-9386-3535205fe52a","Type":"ContainerDied","Data":"957c29d114bfbf9be9af52444f6f1abc571e685cd5c2e5ef3277bb725370beb6"}
Oct 30 19:22:54.665086 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.664800    4372 scope.go:122] "RemoveContainer" containerID="04123e157577e9704e44bfb6f63b69b5f31796ce3c662d0120ff08969dc69b53"
Oct 30 19:22:54.665086 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.664911    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"
Oct 30 19:22:54.680067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.680045    4372 scope.go:122] "RemoveContainer" containerID="04123e157577e9704e44bfb6f63b69b5f31796ce3c662d0120ff08969dc69b53"
Oct 30 19:22:54.680396 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:54.680375    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04123e157577e9704e44bfb6f63b69b5f31796ce3c662d0120ff08969dc69b53\": container with ID starting with 04123e157577e9704e44bfb6f63b69b5f31796ce3c662d0120ff08969dc69b53 not found: ID does not exist" containerID="04123e157577e9704e44bfb6f63b69b5f31796ce3c662d0120ff08969dc69b53"
Oct 30 19:22:54.680481 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.680408    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04123e157577e9704e44bfb6f63b69b5f31796ce3c662d0120ff08969dc69b53"} err="failed to get container status \"04123e157577e9704e44bfb6f63b69b5f31796ce3c662d0120ff08969dc69b53\": rpc error: code = NotFound desc = could not find container \"04123e157577e9704e44bfb6f63b69b5f31796ce3c662d0120ff08969dc69b53\": container with ID starting with 04123e157577e9704e44bfb6f63b69b5f31796ce3c662d0120ff08969dc69b53 not found: ID does not exist"
Oct 30 19:22:54.705675 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.705645    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"]
Oct 30 19:22:54.710843 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:54.710819    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a962a-78c8ffc76f-7vz8l"]
Oct 30 19:22:56.264030 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:56.263991    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4875864-541b-423c-9386-3535205fe52a" path="/var/lib/kubelet/pods/b4875864-541b-423c-9386-3535205fe52a/volumes"
Oct 30 19:22:56.579143 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:56.579093    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"
Oct 30 19:22:56.579332 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:56.579195    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"
Oct 30 19:22:57.422361 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:57.422320    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm" podUID="f4b3db43-1f22-4a7e-84e6-941644be057b" containerName="payload-processing" containerID="cri-o://c1d884b993de777e9e372f43cb9552e53c05c1b9ea3e283b8d0ff51ced32131a" gracePeriod=25
Oct 30 19:22:57.442022 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:57.441973    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv" podUID="6de6da55-2bdc-4cd2-acbe-6892d37ed4c9" containerName="payload-pre-processing" containerID="cri-o://14d220cf348d93dc6619313241946cba9d7019d4e86e8f21bf5e811ca2e00801" gracePeriod=25
Oct 30 19:22:57.695735 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:22:57.695659    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:22:58.644940 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:58.644912    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:22:59.104275 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:22:59.104244    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:23:02.566771 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:02.566735    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"]
Oct 30 19:23:02.567281 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:02.566968    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv" podUID="852083d5-0743-44c0-85fd-0551043be8d7" containerName="istio-proxy" containerID="cri-o://1114d1ae08ad231b03e11a64dc39e3e32071b17c2483c73b41b7b17873318eca" gracePeriod=30
Oct 30 19:23:03.032977 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:03.032930    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm" podUID="f4b3db43-1f22-4a7e-84e6-941644be057b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:03.033225 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:03.032942    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv" podUID="6de6da55-2bdc-4cd2-acbe-6892d37ed4c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:04.254470 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.254433    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"]
Oct 30 19:23:04.254993 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.254940    4372 cpu_manager.go:414] "containerMap: removing container" podUID="b4875864-541b-423c-9386-3535205fe52a" containerName="maas-api"
Oct 30 19:23:04.254993 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.254954    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4875864-541b-423c-9386-3535205fe52a" containerName="maas-api"
Oct 30 19:23:04.255141 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.255046    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4875864-541b-423c-9386-3535205fe52a" containerName="maas-api"
Oct 30 19:23:04.259963 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.259931    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.304361 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.303482    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-58bb38-openshift-default-dockercfg-8bzkr\""
Oct 30 19:23:04.308367 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.308338    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"]
Oct 30 19:23:04.350093 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.350057    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-workload-socket\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.350251 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.350173    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-token\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.350251 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.350206    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istiod-ca-cert\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.350371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.350328    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-envoy\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.350431 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.350383    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-data\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.350495 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.350476    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-workload-certs\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.350539 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.350510    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjjxp\" (UniqueName: \"kubernetes.io/projected/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-kube-api-access-gjjxp\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.350610 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.350593    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-credential-socket\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.350646 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.350635    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-podinfo\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.452874 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.452821    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-workload-socket\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.453196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.452893    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-token\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.453196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.452926    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istiod-ca-cert\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.453196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.452969    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-envoy\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.453196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.453018    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-data\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.453196 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.453091    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-workload-certs\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.454116 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.454078    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istiod-ca-cert\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.454601 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.454572    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-data\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.454775 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.454752    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-workload-certs\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.455227 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.455199    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-workload-socket\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.456848 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.456825    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjjxp\" (UniqueName: \"kubernetes.io/projected/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-kube-api-access-gjjxp\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.457371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.457324    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-credential-socket\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.468563 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.468535    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-podinfo\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.469086 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.468683    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-envoy\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.472589 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.472567    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-podinfo\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.476406 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.476377    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-credential-socket\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.504333 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.504261    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjjxp\" (UniqueName: \"kubernetes.io/projected/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-kube-api-access-gjjxp\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.505375 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.505310    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-token\") pod \"e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") " pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.576343 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.576308    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:04.906345 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.906317    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"]
Oct 30 19:23:04.908053 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:23:04.908018    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e08ca0d_5e22_4a83_b59e_e06c5ec74877.slice/crio-4c790e9b3ffcba4f932d7e06f7ab312d192e1f51c3b07b0cfa94530a68c917be WatchSource:0}: Error finding container 4c790e9b3ffcba4f932d7e06f7ab312d192e1f51c3b07b0cfa94530a68c917be: Status 404 returned error can't find the container with id 4c790e9b3ffcba4f932d7e06f7ab312d192e1f51c3b07b0cfa94530a68c917be
Oct 30 19:23:04.910651 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.910612    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:23:04.910772 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.910696    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:23:04.910772 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:04.910740    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:23:05.740342 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:05.740297    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g" event={"ID":"6e08ca0d-5e22-4a83-b59e-e06c5ec74877","Type":"ContainerStarted","Data":"6eef6f48881c32fa2b977112f8d09949d4d19d6742aa496957f2eeaa94f635f6"}
Oct 30 19:23:05.740748 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:05.740361    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g" event={"ID":"6e08ca0d-5e22-4a83-b59e-e06c5ec74877","Type":"ContainerStarted","Data":"4c790e9b3ffcba4f932d7e06f7ab312d192e1f51c3b07b0cfa94530a68c917be"}
Oct 30 19:23:05.767021 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:05.766960    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g" podStartSLOduration=1.766943284 podStartE2EDuration="1.766943284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:05.764991645 +0000 UTC m=+2999.633409976" watchObservedRunningTime="2026-10-30 19:23:05.766943284 +0000 UTC m=+2999.635361616"
Oct 30 19:23:06.576703 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:06.576501    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:06.583488 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:06.583458    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:06.753148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:06.748010    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:06.753148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:06.749528    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:07.758983 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.758946    4372 generic.go:356] "Generic (PLEG): container finished" podID="852083d5-0743-44c0-85fd-0551043be8d7" containerID="1114d1ae08ad231b03e11a64dc39e3e32071b17c2483c73b41b7b17873318eca" exitCode=0
Oct 30 19:23:07.761299 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.759512    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv" event={"ID":"852083d5-0743-44c0-85fd-0551043be8d7","Type":"ContainerDied","Data":"1114d1ae08ad231b03e11a64dc39e3e32071b17c2483c73b41b7b17873318eca"}
Oct 30 19:23:07.761447 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.761334    4372 scope.go:122] "RemoveContainer" containerID="9e2aa3981d9dff77a42096c37cbc58c4a7c277f468b803c38a4822f1baf8ca9e"
Oct 30 19:23:07.831352 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.831302    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 19:23:07.930741 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.930711    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/852083d5-0743-44c0-85fd-0551043be8d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852083d5-0743-44c0-85fd-0551043be8d7-istio-podinfo\") pod \"852083d5-0743-44c0-85fd-0551043be8d7\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") "
Oct 30 19:23:07.930905 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.930769    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-workload-certs\") pod \"852083d5-0743-44c0-85fd-0551043be8d7\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") "
Oct 30 19:23:07.930905 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.930841    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-istio-envoy\") pod \"852083d5-0743-44c0-85fd-0551043be8d7\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") "
Oct 30 19:23:07.931021 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.930921    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-workload-socket\") pod \"852083d5-0743-44c0-85fd-0551043be8d7\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") "
Oct 30 19:23:07.931021 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.930977    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-istio-data\") pod \"852083d5-0743-44c0-85fd-0551043be8d7\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") "
Oct 30 19:23:07.931021 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.931015    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852083d5-0743-44c0-85fd-0551043be8d7-kube-api-access-kf27p\" (UniqueName: \"kubernetes.io/projected/852083d5-0743-44c0-85fd-0551043be8d7-kube-api-access-kf27p\") pod \"852083d5-0743-44c0-85fd-0551043be8d7\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") "
Oct 30 19:23:07.931238 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.931069    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852083d5-0743-44c0-85fd-0551043be8d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852083d5-0743-44c0-85fd-0551043be8d7-istiod-ca-cert\") pod \"852083d5-0743-44c0-85fd-0551043be8d7\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") "
Oct 30 19:23:07.931238 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.931118    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-workload-certs" pod "852083d5-0743-44c0-85fd-0551043be8d7" (UID: "852083d5-0743-44c0-85fd-0551043be8d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:07.931362 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.931297    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-istio-data" pod "852083d5-0743-44c0-85fd-0551043be8d7" (UID: "852083d5-0743-44c0-85fd-0551043be8d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:07.931551 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.931529    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852083d5-0743-44c0-85fd-0551043be8d7-istio-token\" (UniqueName: \"kubernetes.io/projected/852083d5-0743-44c0-85fd-0551043be8d7-istio-token\") pod \"852083d5-0743-44c0-85fd-0551043be8d7\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") "
Oct 30 19:23:07.931738 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.931709    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-credential-socket\") pod \"852083d5-0743-44c0-85fd-0551043be8d7\" (UID: \"852083d5-0743-44c0-85fd-0551043be8d7\") "
Oct 30 19:23:07.931895 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.931585    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-workload-socket" pod "852083d5-0743-44c0-85fd-0551043be8d7" (UID: "852083d5-0743-44c0-85fd-0551043be8d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:07.932006 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.931530    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852083d5-0743-44c0-85fd-0551043be8d7-istiod-ca-cert" pod "852083d5-0743-44c0-85fd-0551043be8d7" (UID: "852083d5-0743-44c0-85fd-0551043be8d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:07.932151 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.932102    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:07.932266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.932212    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:07.932266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.932230    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:07.932266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.932153    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-credential-socket" pod "852083d5-0743-44c0-85fd-0551043be8d7" (UID: "852083d5-0743-44c0-85fd-0551043be8d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:07.933250 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.933229    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/852083d5-0743-44c0-85fd-0551043be8d7-istio-podinfo" pod "852083d5-0743-44c0-85fd-0551043be8d7" (UID: "852083d5-0743-44c0-85fd-0551043be8d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:23:07.933685 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.933660    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-istio-envoy" pod "852083d5-0743-44c0-85fd-0551043be8d7" (UID: "852083d5-0743-44c0-85fd-0551043be8d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:07.933889 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.933874    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852083d5-0743-44c0-85fd-0551043be8d7-kube-api-access-kf27p" pod "852083d5-0743-44c0-85fd-0551043be8d7" (UID: "852083d5-0743-44c0-85fd-0551043be8d7"). InnerVolumeSpecName "kube-api-access-kf27p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:07.934165 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:07.934105    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852083d5-0743-44c0-85fd-0551043be8d7-istio-token" pod "852083d5-0743-44c0-85fd-0551043be8d7" (UID: "852083d5-0743-44c0-85fd-0551043be8d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:08.033277 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:08.033249    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf27p\" (UniqueName: \"kubernetes.io/projected/852083d5-0743-44c0-85fd-0551043be8d7-kube-api-access-kf27p\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:08.033277 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:08.033274    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852083d5-0743-44c0-85fd-0551043be8d7-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:08.033423 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:08.033286    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852083d5-0743-44c0-85fd-0551043be8d7-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:08.033423 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:08.033296    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:08.033423 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:08.033305    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852083d5-0743-44c0-85fd-0551043be8d7-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:08.033423 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:08.033313    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852083d5-0743-44c0-85fd-0551043be8d7-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:08.768567 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:08.768531    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"
Oct 30 19:23:08.769075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:08.768563    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv" event={"ID":"852083d5-0743-44c0-85fd-0551043be8d7","Type":"ContainerDied","Data":"c114714a383788dcbb5b4522c0b67804990106f9eb711880d5df996b5e3d85f1"}
Oct 30 19:23:08.769075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:08.768636    4372 scope.go:122] "RemoveContainer" containerID="1114d1ae08ad231b03e11a64dc39e3e32071b17c2483c73b41b7b17873318eca"
Oct 30 19:23:08.795516 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:08.795485    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"]
Oct 30 19:23:08.802821 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:08.802796    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a962a-openshift-default-79c77775c5-kfxgv"]
Oct 30 19:23:09.578695 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.578653    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"]
Oct 30 19:23:09.579157 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.579142    4372 cpu_manager.go:414] "containerMap: removing container" podUID="852083d5-0743-44c0-85fd-0551043be8d7" containerName="istio-proxy"
Oct 30 19:23:09.579225 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.579158    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852083d5-0743-44c0-85fd-0551043be8d7" containerName="istio-proxy"
Oct 30 19:23:09.579225 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.579177    4372 cpu_manager.go:414] "containerMap: removing container" podUID="852083d5-0743-44c0-85fd-0551043be8d7" containerName="istio-proxy"
Oct 30 19:23:09.579225 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.579182    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852083d5-0743-44c0-85fd-0551043be8d7" containerName="istio-proxy"
Oct 30 19:23:09.579343 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.579265    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="852083d5-0743-44c0-85fd-0551043be8d7" containerName="istio-proxy"
Oct 30 19:23:09.579343 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.579275    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="852083d5-0743-44c0-85fd-0551043be8d7" containerName="istio-proxy"
Oct 30 19:23:09.582523 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.582503    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"
Oct 30 19:23:09.585978 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.585957    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"]
Oct 30 19:23:09.590020 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.589998    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"
Oct 30 19:23:09.598800 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.598775    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-58bb38\""
Oct 30 19:23:09.598993 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.598974    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-58bb38-dockercfg-4twnf\""
Oct 30 19:23:09.599526 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.599502    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"]
Oct 30 19:23:09.602782 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.602265    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"]
Oct 30 19:23:09.753821 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.753784    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p78ph\" (UniqueName: \"kubernetes.io/projected/d49ae273-a4bf-4aae-bbac-6ed7c086f024-kube-api-access-p78ph\") pod \"payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m\" (UID: \"d49ae273-a4bf-4aae-bbac-6ed7c086f024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"
Oct 30 19:23:09.754021 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.753996    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d49ae273-a4bf-4aae-bbac-6ed7c086f024-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m\" (UID: \"d49ae273-a4bf-4aae-bbac-6ed7c086f024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"
Oct 30 19:23:09.754082 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.754058    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn\" (UID: \"34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"
Oct 30 19:23:09.754251 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.754232    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b928\" (UniqueName: \"kubernetes.io/projected/34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665-kube-api-access-6b928\") pod \"payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn\" (UID: \"34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"
Oct 30 19:23:09.777257 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.777228    4372 generic.go:356] "Generic (PLEG): container finished" podID="450a9e2f-36a4-419e-a586-981cda482d97" containerID="a733c275f1c2935a125d0ebcd73118214a3d414c342f9db476ea702321d5e396" exitCode=0
Oct 30 19:23:09.777710 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.777303    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v" event={"ID":"450a9e2f-36a4-419e-a586-981cda482d97","Type":"ContainerDied","Data":"a733c275f1c2935a125d0ebcd73118214a3d414c342f9db476ea702321d5e396"}
Oct 30 19:23:09.777832 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.777813    4372 scope.go:122] "RemoveContainer" containerID="a733c275f1c2935a125d0ebcd73118214a3d414c342f9db476ea702321d5e396"
Oct 30 19:23:09.779482 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.779445    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:23:09.779609 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.779527    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:23:09.779609 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.779568    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:23:09.856455 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.855802    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d49ae273-a4bf-4aae-bbac-6ed7c086f024-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m\" (UID: \"d49ae273-a4bf-4aae-bbac-6ed7c086f024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"
Oct 30 19:23:09.856455 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.855871    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn\" (UID: \"34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"
Oct 30 19:23:09.856455 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.855960    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b928\" (UniqueName: \"kubernetes.io/projected/34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665-kube-api-access-6b928\") pod \"payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn\" (UID: \"34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"
Oct 30 19:23:09.856455 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.856005    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p78ph\" (UniqueName: \"kubernetes.io/projected/d49ae273-a4bf-4aae-bbac-6ed7c086f024-kube-api-access-p78ph\") pod \"payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m\" (UID: \"d49ae273-a4bf-4aae-bbac-6ed7c086f024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"
Oct 30 19:23:09.857066 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.857039    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d49ae273-a4bf-4aae-bbac-6ed7c086f024-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m\" (UID: \"d49ae273-a4bf-4aae-bbac-6ed7c086f024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"
Oct 30 19:23:09.857245 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.857074    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn\" (UID: \"34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"
Oct 30 19:23:09.870165 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.870116    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b928\" (UniqueName: \"kubernetes.io/projected/34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665-kube-api-access-6b928\") pod \"payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn\" (UID: \"34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"
Oct 30 19:23:09.874066 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.874040    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p78ph\" (UniqueName: \"kubernetes.io/projected/d49ae273-a4bf-4aae-bbac-6ed7c086f024-kube-api-access-p78ph\") pod \"payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m\" (UID: \"d49ae273-a4bf-4aae-bbac-6ed7c086f024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"
Oct 30 19:23:09.897762 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.897737    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"
Oct 30 19:23:09.912630 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:09.911264    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"
Oct 30 19:23:10.201420 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:10.201394    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"]
Oct 30 19:23:10.202791 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:23:10.202764    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ce7f8a_dc9d_4b3e_b75d_a5d0f665b665.slice/crio-7e59235f1b660358a560aa0f05aa1897680f195f46ad04a03d62df4fcceabd8c WatchSource:0}: Error finding container 7e59235f1b660358a560aa0f05aa1897680f195f46ad04a03d62df4fcceabd8c: Status 404 returned error can't find the container with id 7e59235f1b660358a560aa0f05aa1897680f195f46ad04a03d62df4fcceabd8c
Oct 30 19:23:10.209164 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:10.209142    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"]
Oct 30 19:23:10.211664 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:23:10.211629    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49ae273_a4bf_4aae_bbac_6ed7c086f024.slice/crio-286c4d0e8d72d0947b58cf1a0f468f308a0703839e7382739c6af107cb8fa70b WatchSource:0}: Error finding container 286c4d0e8d72d0947b58cf1a0f468f308a0703839e7382739c6af107cb8fa70b: Status 404 returned error can't find the container with id 286c4d0e8d72d0947b58cf1a0f468f308a0703839e7382739c6af107cb8fa70b
Oct 30 19:23:10.270801 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:10.270771    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852083d5-0743-44c0-85fd-0551043be8d7" path="/var/lib/kubelet/pods/852083d5-0743-44c0-85fd-0551043be8d7/volumes"
Oct 30 19:23:10.794409 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:10.794369    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v" event={"ID":"450a9e2f-36a4-419e-a586-981cda482d97","Type":"ContainerStarted","Data":"f5ffdb0cb85780d7db97c2a1c2bf4a6ca2b8e75afb082ae7338800c0dd4d7b5f"}
Oct 30 19:23:10.798067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:10.797952    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn" event={"ID":"34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665","Type":"ContainerStarted","Data":"63378dd8fdbfd98bdc25df977fc8591ab54c3cc5d25962d19bce8c9887ef12c6"}
Oct 30 19:23:10.798067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:10.798005    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn" event={"ID":"34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665","Type":"ContainerStarted","Data":"7e59235f1b660358a560aa0f05aa1897680f195f46ad04a03d62df4fcceabd8c"}
Oct 30 19:23:10.800202 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:10.800057    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m" event={"ID":"d49ae273-a4bf-4aae-bbac-6ed7c086f024","Type":"ContainerStarted","Data":"42b1561c2e5e2224c93a3f530c77cecc6d00bf7915d7694111a89c8117aec5a2"}
Oct 30 19:23:10.800202 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:10.800103    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m" event={"ID":"d49ae273-a4bf-4aae-bbac-6ed7c086f024","Type":"ContainerStarted","Data":"286c4d0e8d72d0947b58cf1a0f468f308a0703839e7382739c6af107cb8fa70b"}
Oct 30 19:23:10.801347 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:10.801327    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"
Oct 30 19:23:10.802683 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:10.802665    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"
Oct 30 19:23:10.847505 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:10.847449    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m" podStartSLOduration=1.8474370260000001 podStartE2EDuration="1.847437026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:10.845094293 +0000 UTC m=+3004.713512623" watchObservedRunningTime="2026-10-30 19:23:10.847437026 +0000 UTC m=+3004.715855355"
Oct 30 19:23:10.865826 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:10.865779    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn" podStartSLOduration=1.8657676699999999 podStartE2EDuration="1.86576767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:10.863882865 +0000 UTC m=+3004.732301195" watchObservedRunningTime="2026-10-30 19:23:10.86576767 +0000 UTC m=+3004.734185994"
Oct 30 19:23:11.454282 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:11.454247    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:23:11.455334 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:11.455308    4372 patch_prober.go:29] interesting pod/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 19:23:11.455468 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:11.455365    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v" podUID="450a9e2f-36a4-419e-a586-981cda482d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 19:23:11.537106 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:11.537078    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:23:12.454665 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:12.454628    4372 patch_prober.go:29] interesting pod/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 19:23:12.455100 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:12.454682    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v" podUID="450a9e2f-36a4-419e-a586-981cda482d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 19:23:13.031891 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:13.031850    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv" podUID="6de6da55-2bdc-4cd2-acbe-6892d37ed4c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:13.031891 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:13.031870    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm" podUID="f4b3db43-1f22-4a7e-84e6-941644be057b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:13.453930 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:13.453896    4372 patch_prober.go:29] interesting pod/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 19:23:13.454119 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:13.453950    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v" podUID="450a9e2f-36a4-419e-a586-981cda482d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 19:23:14.454471 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:14.454433    4372 patch_prober.go:29] interesting pod/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 19:23:14.454870 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:14.454487    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v" podUID="450a9e2f-36a4-419e-a586-981cda482d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 19:23:15.454012 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.453974    4372 patch_prober.go:29] interesting pod/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 19:23:15.454218 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.454024    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v" podUID="450a9e2f-36a4-419e-a586-981cda482d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 19:23:15.628118 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.628057    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"]
Oct 30 19:23:15.632778 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.632748    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:15.644564 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.643030    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-58bb38-serving-cert\""
Oct 30 19:23:15.651083 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.651050    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"]
Oct 30 19:23:15.775657 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.775553    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8144c849-ae43-4536-b9d1-4dbdde207847-openshift-service-ca\") pod \"maas-api-e2e-inf-b-58bb38-769587d694-hwg7r\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:15.776141 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.776091    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79cqf\" (UniqueName: \"kubernetes.io/projected/8144c849-ae43-4536-b9d1-4dbdde207847-kube-api-access-79cqf\") pod \"maas-api-e2e-inf-b-58bb38-769587d694-hwg7r\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:15.788481 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.776155    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8144c849-ae43-4536-b9d1-4dbdde207847-maas-api-tls\") pod \"maas-api-e2e-inf-b-58bb38-769587d694-hwg7r\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:15.788481 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.787805    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8144c849-ae43-4536-b9d1-4dbdde207847-metrics-certs\") pod \"maas-api-e2e-inf-b-58bb38-769587d694-hwg7r\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:15.889042 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.889008    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79cqf\" (UniqueName: \"kubernetes.io/projected/8144c849-ae43-4536-b9d1-4dbdde207847-kube-api-access-79cqf\") pod \"maas-api-e2e-inf-b-58bb38-769587d694-hwg7r\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:15.889042 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.889045    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8144c849-ae43-4536-b9d1-4dbdde207847-maas-api-tls\") pod \"maas-api-e2e-inf-b-58bb38-769587d694-hwg7r\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:15.889266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.889108    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8144c849-ae43-4536-b9d1-4dbdde207847-metrics-certs\") pod \"maas-api-e2e-inf-b-58bb38-769587d694-hwg7r\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:15.889266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.889170    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8144c849-ae43-4536-b9d1-4dbdde207847-openshift-service-ca\") pod \"maas-api-e2e-inf-b-58bb38-769587d694-hwg7r\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:15.889873 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.889796    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8144c849-ae43-4536-b9d1-4dbdde207847-openshift-service-ca\") pod \"maas-api-e2e-inf-b-58bb38-769587d694-hwg7r\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:15.891730 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.891706    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8144c849-ae43-4536-b9d1-4dbdde207847-maas-api-tls\") pod \"maas-api-e2e-inf-b-58bb38-769587d694-hwg7r\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:15.891871 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.891848    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8144c849-ae43-4536-b9d1-4dbdde207847-metrics-certs\") pod \"maas-api-e2e-inf-b-58bb38-769587d694-hwg7r\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:15.902716 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.902689    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79cqf\" (UniqueName: \"kubernetes.io/projected/8144c849-ae43-4536-b9d1-4dbdde207847-kube-api-access-79cqf\") pod \"maas-api-e2e-inf-b-58bb38-769587d694-hwg7r\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:15.962380 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:15.962357    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:16.251991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.251960    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"]
Oct 30 19:23:16.252545 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:23:16.252523    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8144c849_ae43_4536_b9d1_4dbdde207847.slice/crio-b2e7b500a5e2b86ec56d623a958411451d034022cf219f43ef19ec2212842b87 WatchSource:0}: Error finding container b2e7b500a5e2b86ec56d623a958411451d034022cf219f43ef19ec2212842b87: Status 404 returned error can't find the container with id b2e7b500a5e2b86ec56d623a958411451d034022cf219f43ef19ec2212842b87
Oct 30 19:23:16.458891 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.458849    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:23:16.460156 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.460100    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:23:16.478304 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.478275    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"]
Oct 30 19:23:16.478534 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.478499    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh" podUID="18722d33-fb04-43df-bf56-3722c86e6346" containerName="maas-api" containerID="cri-o://89a4134d2acb27d824964d360250450948d131baf8768f97624c33757806c4c2" gracePeriod=30
Oct 30 19:23:16.791325 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.791274    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 19:23:16.861002 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.860909    4372 generic.go:356] "Generic (PLEG): container finished" podID="18722d33-fb04-43df-bf56-3722c86e6346" containerID="89a4134d2acb27d824964d360250450948d131baf8768f97624c33757806c4c2" exitCode=0
Oct 30 19:23:16.861002 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.860963    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"
Oct 30 19:23:16.861255 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.860997    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh" event={"ID":"18722d33-fb04-43df-bf56-3722c86e6346","Type":"ContainerDied","Data":"89a4134d2acb27d824964d360250450948d131baf8768f97624c33757806c4c2"}
Oct 30 19:23:16.861255 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.861064    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh" event={"ID":"18722d33-fb04-43df-bf56-3722c86e6346","Type":"ContainerDied","Data":"d5430e9c190ddf577a9d7fdebd16e6e6feeca2626e33e4d0c466c1894c826caf"}
Oct 30 19:23:16.861255 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.861097    4372 scope.go:122] "RemoveContainer" containerID="89a4134d2acb27d824964d360250450948d131baf8768f97624c33757806c4c2"
Oct 30 19:23:16.862895 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.862869    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r" event={"ID":"8144c849-ae43-4536-b9d1-4dbdde207847","Type":"ContainerStarted","Data":"c1b29206cd9dbc4e15056d9a195fda2e1b5eb117c06d4f6154d3044623f466fa"}
Oct 30 19:23:16.863044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.863026    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:16.863165 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.863073    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r" event={"ID":"8144c849-ae43-4536-b9d1-4dbdde207847","Type":"ContainerStarted","Data":"b2e7b500a5e2b86ec56d623a958411451d034022cf219f43ef19ec2212842b87"}
Oct 30 19:23:16.870903 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.870885    4372 scope.go:122] "RemoveContainer" containerID="89a4134d2acb27d824964d360250450948d131baf8768f97624c33757806c4c2"
Oct 30 19:23:16.871196 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:23:16.871173    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a4134d2acb27d824964d360250450948d131baf8768f97624c33757806c4c2\": container with ID starting with 89a4134d2acb27d824964d360250450948d131baf8768f97624c33757806c4c2 not found: ID does not exist" containerID="89a4134d2acb27d824964d360250450948d131baf8768f97624c33757806c4c2"
Oct 30 19:23:16.871265 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.871203    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a4134d2acb27d824964d360250450948d131baf8768f97624c33757806c4c2"} err="failed to get container status \"89a4134d2acb27d824964d360250450948d131baf8768f97624c33757806c4c2\": rpc error: code = NotFound desc = could not find container \"89a4134d2acb27d824964d360250450948d131baf8768f97624c33757806c4c2\": container with ID starting with 89a4134d2acb27d824964d360250450948d131baf8768f97624c33757806c4c2 not found: ID does not exist"
Oct 30 19:23:16.892277 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.892236    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r" podStartSLOduration=1.4412302399999999 podStartE2EDuration="1.892223941s" totalImagesPullingTime="450.993701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:16.889662362 +0000 UTC m=+3010.758080692" watchObservedRunningTime="2026-10-30 19:23:16.892223941 +0000 UTC m=+3010.760642271"
Oct 30 19:23:16.898402 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.898375    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18722d33-fb04-43df-bf56-3722c86e6346-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18722d33-fb04-43df-bf56-3722c86e6346-openshift-service-ca\") pod \"18722d33-fb04-43df-bf56-3722c86e6346\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") "
Oct 30 19:23:16.898517 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.898421    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18722d33-fb04-43df-bf56-3722c86e6346-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18722d33-fb04-43df-bf56-3722c86e6346-maas-api-tls\") pod \"18722d33-fb04-43df-bf56-3722c86e6346\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") "
Oct 30 19:23:16.898578 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.898564    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18722d33-fb04-43df-bf56-3722c86e6346-kube-api-access-jk5mj\" (UniqueName: \"kubernetes.io/projected/18722d33-fb04-43df-bf56-3722c86e6346-kube-api-access-jk5mj\") pod \"18722d33-fb04-43df-bf56-3722c86e6346\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") "
Oct 30 19:23:16.898650 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.898635    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18722d33-fb04-43df-bf56-3722c86e6346-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18722d33-fb04-43df-bf56-3722c86e6346-metrics-certs\") pod \"18722d33-fb04-43df-bf56-3722c86e6346\" (UID: \"18722d33-fb04-43df-bf56-3722c86e6346\") "
Oct 30 19:23:16.898797 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.898773    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18722d33-fb04-43df-bf56-3722c86e6346-openshift-service-ca" pod "18722d33-fb04-43df-bf56-3722c86e6346" (UID: "18722d33-fb04-43df-bf56-3722c86e6346"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:16.899704 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.899672    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18722d33-fb04-43df-bf56-3722c86e6346-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:16.901287 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.901206    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18722d33-fb04-43df-bf56-3722c86e6346-metrics-certs" pod "18722d33-fb04-43df-bf56-3722c86e6346" (UID: "18722d33-fb04-43df-bf56-3722c86e6346"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:16.901755 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.901670    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18722d33-fb04-43df-bf56-3722c86e6346-kube-api-access-jk5mj" pod "18722d33-fb04-43df-bf56-3722c86e6346" (UID: "18722d33-fb04-43df-bf56-3722c86e6346"). InnerVolumeSpecName "kube-api-access-jk5mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:16.901974 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:16.901950    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18722d33-fb04-43df-bf56-3722c86e6346-maas-api-tls" pod "18722d33-fb04-43df-bf56-3722c86e6346" (UID: "18722d33-fb04-43df-bf56-3722c86e6346"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:17.000239 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.000181    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18722d33-fb04-43df-bf56-3722c86e6346-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:17.000239 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.000206    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk5mj\" (UniqueName: \"kubernetes.io/projected/18722d33-fb04-43df-bf56-3722c86e6346-kube-api-access-jk5mj\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:17.000239 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.000216    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18722d33-fb04-43df-bf56-3722c86e6346-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:17.192158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.192111    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"]
Oct 30 19:23:17.200108 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.200070    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ab7df0-695f77dc59-dmrkh"]
Oct 30 19:23:17.672309 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.672281    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"
Oct 30 19:23:17.684482 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.684433    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"
Oct 30 19:23:17.706901 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.706870    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de6da55-2bdc-4cd2-acbe-6892d37ed4c9-kube-api-access-nz4zt\" (UniqueName: \"kubernetes.io/projected/6de6da55-2bdc-4cd2-acbe-6892d37ed4c9-kube-api-access-nz4zt\") pod \"6de6da55-2bdc-4cd2-acbe-6892d37ed4c9\" (UID: \"6de6da55-2bdc-4cd2-acbe-6892d37ed4c9\") "
Oct 30 19:23:17.707043 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.706925    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6de6da55-2bdc-4cd2-acbe-6892d37ed4c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de6da55-2bdc-4cd2-acbe-6892d37ed4c9-plugins-config-volume\") pod \"6de6da55-2bdc-4cd2-acbe-6892d37ed4c9\" (UID: \"6de6da55-2bdc-4cd2-acbe-6892d37ed4c9\") "
Oct 30 19:23:17.707117 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.707071    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b3db43-1f22-4a7e-84e6-941644be057b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b3db43-1f22-4a7e-84e6-941644be057b-plugins-config-volume\") pod \"f4b3db43-1f22-4a7e-84e6-941644be057b\" (UID: \"f4b3db43-1f22-4a7e-84e6-941644be057b\") "
Oct 30 19:23:17.707117 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.707096    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b3db43-1f22-4a7e-84e6-941644be057b-kube-api-access-9bn2j\" (UniqueName: \"kubernetes.io/projected/f4b3db43-1f22-4a7e-84e6-941644be057b-kube-api-access-9bn2j\") pod \"f4b3db43-1f22-4a7e-84e6-941644be057b\" (UID: \"f4b3db43-1f22-4a7e-84e6-941644be057b\") "
Oct 30 19:23:17.707477 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.707445    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de6da55-2bdc-4cd2-acbe-6892d37ed4c9-plugins-config-volume" pod "6de6da55-2bdc-4cd2-acbe-6892d37ed4c9" (UID: "6de6da55-2bdc-4cd2-acbe-6892d37ed4c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:17.707577 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.707469    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b3db43-1f22-4a7e-84e6-941644be057b-plugins-config-volume" pod "f4b3db43-1f22-4a7e-84e6-941644be057b" (UID: "f4b3db43-1f22-4a7e-84e6-941644be057b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:17.709159 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.709113    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de6da55-2bdc-4cd2-acbe-6892d37ed4c9-kube-api-access-nz4zt" pod "6de6da55-2bdc-4cd2-acbe-6892d37ed4c9" (UID: "6de6da55-2bdc-4cd2-acbe-6892d37ed4c9"). InnerVolumeSpecName "kube-api-access-nz4zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:17.709431 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.709405    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b3db43-1f22-4a7e-84e6-941644be057b-kube-api-access-9bn2j" pod "f4b3db43-1f22-4a7e-84e6-941644be057b" (UID: "f4b3db43-1f22-4a7e-84e6-941644be057b"). InnerVolumeSpecName "kube-api-access-9bn2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:17.808392 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.808368    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de6da55-2bdc-4cd2-acbe-6892d37ed4c9-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:17.808392 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.808392    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b3db43-1f22-4a7e-84e6-941644be057b-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:17.808775 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.808401    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bn2j\" (UniqueName: \"kubernetes.io/projected/f4b3db43-1f22-4a7e-84e6-941644be057b-kube-api-access-9bn2j\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:17.808775 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.808410    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz4zt\" (UniqueName: \"kubernetes.io/projected/6de6da55-2bdc-4cd2-acbe-6892d37ed4c9-kube-api-access-nz4zt\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:17.869864 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.869838    4372 generic.go:356] "Generic (PLEG): container finished" podID="6de6da55-2bdc-4cd2-acbe-6892d37ed4c9" containerID="14d220cf348d93dc6619313241946cba9d7019d4e86e8f21bf5e811ca2e00801" exitCode=0
Oct 30 19:23:17.869964 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.869882    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"
Oct 30 19:23:17.869964 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.869916    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv" event={"ID":"6de6da55-2bdc-4cd2-acbe-6892d37ed4c9","Type":"ContainerDied","Data":"14d220cf348d93dc6619313241946cba9d7019d4e86e8f21bf5e811ca2e00801"}
Oct 30 19:23:17.870081 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.869977    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv" event={"ID":"6de6da55-2bdc-4cd2-acbe-6892d37ed4c9","Type":"ContainerDied","Data":"9a8ee020a6bd609a5f74ac41f6e26c0a53b27907fa1255afab82f8ac8af5fc64"}
Oct 30 19:23:17.870081 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.870009    4372 scope.go:122] "RemoveContainer" containerID="14d220cf348d93dc6619313241946cba9d7019d4e86e8f21bf5e811ca2e00801"
Oct 30 19:23:17.871398 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.871379    4372 generic.go:356] "Generic (PLEG): container finished" podID="f4b3db43-1f22-4a7e-84e6-941644be057b" containerID="c1d884b993de777e9e372f43cb9552e53c05c1b9ea3e283b8d0ff51ced32131a" exitCode=0
Oct 30 19:23:17.871494 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.871408    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm" event={"ID":"f4b3db43-1f22-4a7e-84e6-941644be057b","Type":"ContainerDied","Data":"c1d884b993de777e9e372f43cb9552e53c05c1b9ea3e283b8d0ff51ced32131a"}
Oct 30 19:23:17.871494 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.871421    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"
Oct 30 19:23:17.871494 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.871458    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm" event={"ID":"f4b3db43-1f22-4a7e-84e6-941644be057b","Type":"ContainerDied","Data":"eba6bd3cfb456e027df21012d7005d8ba52c817d89c22d6fb70b595b8b8a72a8"}
Oct 30 19:23:17.881438 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.881423    4372 scope.go:122] "RemoveContainer" containerID="14d220cf348d93dc6619313241946cba9d7019d4e86e8f21bf5e811ca2e00801"
Oct 30 19:23:17.881753 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:23:17.881731    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d220cf348d93dc6619313241946cba9d7019d4e86e8f21bf5e811ca2e00801\": container with ID starting with 14d220cf348d93dc6619313241946cba9d7019d4e86e8f21bf5e811ca2e00801 not found: ID does not exist" containerID="14d220cf348d93dc6619313241946cba9d7019d4e86e8f21bf5e811ca2e00801"
Oct 30 19:23:17.881845 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.881761    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d220cf348d93dc6619313241946cba9d7019d4e86e8f21bf5e811ca2e00801"} err="failed to get container status \"14d220cf348d93dc6619313241946cba9d7019d4e86e8f21bf5e811ca2e00801\": rpc error: code = NotFound desc = could not find container \"14d220cf348d93dc6619313241946cba9d7019d4e86e8f21bf5e811ca2e00801\": container with ID starting with 14d220cf348d93dc6619313241946cba9d7019d4e86e8f21bf5e811ca2e00801 not found: ID does not exist"
Oct 30 19:23:17.881845 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.881784    4372 scope.go:122] "RemoveContainer" containerID="c1d884b993de777e9e372f43cb9552e53c05c1b9ea3e283b8d0ff51ced32131a"
Oct 30 19:23:17.890433 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.890324    4372 scope.go:122] "RemoveContainer" containerID="c1d884b993de777e9e372f43cb9552e53c05c1b9ea3e283b8d0ff51ced32131a"
Oct 30 19:23:17.890658 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:23:17.890638    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d884b993de777e9e372f43cb9552e53c05c1b9ea3e283b8d0ff51ced32131a\": container with ID starting with c1d884b993de777e9e372f43cb9552e53c05c1b9ea3e283b8d0ff51ced32131a not found: ID does not exist" containerID="c1d884b993de777e9e372f43cb9552e53c05c1b9ea3e283b8d0ff51ced32131a"
Oct 30 19:23:17.890721 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.890665    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d884b993de777e9e372f43cb9552e53c05c1b9ea3e283b8d0ff51ced32131a"} err="failed to get container status \"c1d884b993de777e9e372f43cb9552e53c05c1b9ea3e283b8d0ff51ced32131a\": rpc error: code = NotFound desc = could not find container \"c1d884b993de777e9e372f43cb9552e53c05c1b9ea3e283b8d0ff51ced32131a\": container with ID starting with c1d884b993de777e9e372f43cb9552e53c05c1b9ea3e283b8d0ff51ced32131a not found: ID does not exist"
Oct 30 19:23:17.904463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.904440    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"]
Oct 30 19:23:17.911253 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.911111    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a962a-96f6786f9-qf2hv"]
Oct 30 19:23:17.923052 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.923030    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"]
Oct 30 19:23:17.925251 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:17.925232    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a962a-68fbb7bdc6-k84sm"]
Oct 30 19:23:18.265059 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:18.265028    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18722d33-fb04-43df-bf56-3722c86e6346" path="/var/lib/kubelet/pods/18722d33-fb04-43df-bf56-3722c86e6346/volumes"
Oct 30 19:23:18.265504 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:18.265489    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de6da55-2bdc-4cd2-acbe-6892d37ed4c9" path="/var/lib/kubelet/pods/6de6da55-2bdc-4cd2-acbe-6892d37ed4c9/volumes"
Oct 30 19:23:18.265848 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:18.265836    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b3db43-1f22-4a7e-84e6-941644be057b" path="/var/lib/kubelet/pods/f4b3db43-1f22-4a7e-84e6-941644be057b/volumes"
Oct 30 19:23:22.817275 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:22.817239    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"
Oct 30 19:23:22.817680 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:22.817435    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"
Oct 30 19:23:22.879120 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:22.879090    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:23:23.697297 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:23.697263    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:23:27.714351 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:23:27.714317    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:23:30.060951 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.060916    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"]
Oct 30 19:23:30.062840 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.062811    4372 cpu_manager.go:414] "containerMap: removing container" podUID="6de6da55-2bdc-4cd2-acbe-6892d37ed4c9" containerName="payload-pre-processing"
Oct 30 19:23:30.062840 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.062840    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de6da55-2bdc-4cd2-acbe-6892d37ed4c9" containerName="payload-pre-processing"
Oct 30 19:23:30.063068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.062867    4372 cpu_manager.go:414] "containerMap: removing container" podUID="f4b3db43-1f22-4a7e-84e6-941644be057b" containerName="payload-processing"
Oct 30 19:23:30.063068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.062874    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b3db43-1f22-4a7e-84e6-941644be057b" containerName="payload-processing"
Oct 30 19:23:30.063068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.062924    4372 cpu_manager.go:414] "containerMap: removing container" podUID="18722d33-fb04-43df-bf56-3722c86e6346" containerName="maas-api"
Oct 30 19:23:30.063068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.062932    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18722d33-fb04-43df-bf56-3722c86e6346" containerName="maas-api"
Oct 30 19:23:30.063068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.063053    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="18722d33-fb04-43df-bf56-3722c86e6346" containerName="maas-api"
Oct 30 19:23:30.063366 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.063070    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de6da55-2bdc-4cd2-acbe-6892d37ed4c9" containerName="payload-pre-processing"
Oct 30 19:23:30.063366 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.063081    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b3db43-1f22-4a7e-84e6-941644be057b" containerName="payload-processing"
Oct 30 19:23:30.068116 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.068097    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.073207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.073183    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fe8ece\"/\"default-dockercfg-vfqw5\""
Oct 30 19:23:30.076237 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.076210    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fe8ece\"/\"openshift-service-ca.crt\""
Oct 30 19:23:30.076479 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.076457    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fe8ece\"/\"test-model-fe8ece-kserve-self-signed-certs\""
Oct 30 19:23:30.103149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.102544    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fe8ece\"/\"kube-root-ca.crt\""
Oct 30 19:23:30.110311 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.110278    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"]
Oct 30 19:23:30.172388 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.172348    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-model-cache\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.172541 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.172399    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4kpn\" (UniqueName: \"kubernetes.io/projected/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-kube-api-access-c4kpn\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.172541 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.172509    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-home\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.172726 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.172707    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-tmp-dir\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.172783 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.172744    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-dshm\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.172783 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.172775    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-tls-certs\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.275952 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.275916    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-tmp-dir\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.276252 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.276230    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-dshm\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.278184 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.277824    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-tls-certs\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.278184 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.277869    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-model-cache\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.278184 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.277920    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4kpn\" (UniqueName: \"kubernetes.io/projected/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-kube-api-access-c4kpn\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.278184 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.277989    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-home\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.282152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.278499    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-home\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.282152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.278767    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-model-cache\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.282152 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.276446    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-tmp-dir\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.313197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.312813    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4kpn\" (UniqueName: \"kubernetes.io/projected/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-kube-api-access-c4kpn\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.316461 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.316429    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-dshm\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.325762 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.325735    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-tls-certs\") pod \"test-model-fe8ece-kserve-595c7859f-b2tsb\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") " pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.407168 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.407099    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:30.686740 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.686710    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"]
Oct 30 19:23:30.688498 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:23:30.688471    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba2afdc1_190a_4aea_9cc9_2692bcaf56c7.slice/crio-02d03539b2a4b35dd59884b5266237efc6b8f46a17ae9382b2f884c13fbe551a WatchSource:0}: Error finding container 02d03539b2a4b35dd59884b5266237efc6b8f46a17ae9382b2f884c13fbe551a: Status 404 returned error can't find the container with id 02d03539b2a4b35dd59884b5266237efc6b8f46a17ae9382b2f884c13fbe551a
Oct 30 19:23:30.949595 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.949511    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb" event={"ID":"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7","Type":"ContainerStarted","Data":"14ce2d8176b1bb3b101ce4bdfd07100c3ca3374ad45f46a93931f9d8267496f6"}
Oct 30 19:23:30.949595 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.949558    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb" event={"ID":"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7","Type":"ContainerStarted","Data":"02d03539b2a4b35dd59884b5266237efc6b8f46a17ae9382b2f884c13fbe551a"}
Oct 30 19:23:30.971991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:30.971942    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb" podStartSLOduration=0.971928641 podStartE2EDuration="971.928641ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:30.969999633 +0000 UTC m=+3024.838417966" watchObservedRunningTime="2026-10-30 19:23:30.971928641 +0000 UTC m=+3024.840346998"
Oct 30 19:23:31.407527 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:31.407494    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:31.654834 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:31.654795    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"]
Oct 30 19:23:31.795302 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:31.795218    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"]
Oct 30 19:23:31.799931 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:31.799905    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:31.804403 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:31.804366    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-58bb38\"/\"default-dockercfg-h9mzm\""
Oct 30 19:23:31.804931 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:31.804901    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-58bb38\"/\"kube-root-ca.crt\""
Oct 30 19:23:31.805039 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:31.804979    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-58bb38\"/\"openshift-service-ca.crt\""
Oct 30 19:23:31.808299 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:31.808276    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-58bb38\"/\"test-model-58bb38-kserve-self-signed-certs\""
Oct 30 19:23:31.813274 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:31.813247    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"]
Oct 30 19:23:31.911951 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:31.911914    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-tmp-dir\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:31.912158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:31.911979    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-home\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:31.912158 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:31.912149    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgfmk\" (UniqueName: \"kubernetes.io/projected/29099d2d-89eb-49da-b9ea-7e9c4914d571-kube-api-access-hgfmk\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:31.912283 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:31.912207    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-dshm\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:31.912283 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:31.912225    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29099d2d-89eb-49da-b9ea-7e9c4914d571-tls-certs\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:31.912364 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:31.912298    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-model-cache\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:32.014058 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.014021    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfmk\" (UniqueName: \"kubernetes.io/projected/29099d2d-89eb-49da-b9ea-7e9c4914d571-kube-api-access-hgfmk\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:32.014268 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.014095    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-dshm\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:32.014268 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.014141    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29099d2d-89eb-49da-b9ea-7e9c4914d571-tls-certs\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:32.014268 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.014165    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-model-cache\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:32.014268 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.014244    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-tmp-dir\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:32.014532 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.014306    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-home\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:32.014654 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.014631    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-model-cache\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:32.014720 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.014673    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-home\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:32.014824 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.014804    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-tmp-dir\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:32.016961 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.016933    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29099d2d-89eb-49da-b9ea-7e9c4914d571-tls-certs\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:32.019157 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.017710    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-dshm\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:32.059198 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.059076    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgfmk\" (UniqueName: \"kubernetes.io/projected/29099d2d-89eb-49da-b9ea-7e9c4914d571-kube-api-access-hgfmk\") pod \"test-model-58bb38-kserve-7c7bf94f5f-vx5tk\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") " pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:32.116198 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.116156    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:32.411537 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.411496    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"]
Oct 30 19:23:32.413185 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:23:32.413152    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29099d2d_89eb_49da_b9ea_7e9c4914d571.slice/crio-9bfcf0d5035d931e62d8566132be572702af19d34b1b17258264ad42e8d10bf3 WatchSource:0}: Error finding container 9bfcf0d5035d931e62d8566132be572702af19d34b1b17258264ad42e8d10bf3: Status 404 returned error can't find the container with id 9bfcf0d5035d931e62d8566132be572702af19d34b1b17258264ad42e8d10bf3
Oct 30 19:23:32.963448 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.963407    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk" event={"ID":"29099d2d-89eb-49da-b9ea-7e9c4914d571","Type":"ContainerStarted","Data":"3aaacc29dba697399719addb7a25c2760a00dee583a22bb8166004e541a2e751"}
Oct 30 19:23:32.963632 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.963457    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk" event={"ID":"29099d2d-89eb-49da-b9ea-7e9c4914d571","Type":"ContainerStarted","Data":"9bfcf0d5035d931e62d8566132be572702af19d34b1b17258264ad42e8d10bf3"}
Oct 30 19:23:32.988798 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:32.988630    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk" podStartSLOduration=1.988612744 podStartE2EDuration="1.988612744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:32.988446959 +0000 UTC m=+3026.856865291" watchObservedRunningTime="2026-10-30 19:23:32.988612744 +0000 UTC m=+3026.857031075"
Oct 30 19:23:33.117844 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:33.117808    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:34.975568 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:34.975538    4372 generic.go:356] "Generic (PLEG): container finished" podID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" containerID="6eef6f48881c32fa2b977112f8d09949d4d19d6742aa496957f2eeaa94f635f6" exitCode=0
Oct 30 19:23:34.976058 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:34.975621    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g" event={"ID":"6e08ca0d-5e22-4a83-b59e-e06c5ec74877","Type":"ContainerDied","Data":"6eef6f48881c32fa2b977112f8d09949d4d19d6742aa496957f2eeaa94f635f6"}
Oct 30 19:23:34.976107 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:34.976071    4372 scope.go:122] "RemoveContainer" containerID="6eef6f48881c32fa2b977112f8d09949d4d19d6742aa496957f2eeaa94f635f6"
Oct 30 19:23:34.977522 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:34.977493    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:23:34.977606 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:34.977567    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:23:34.977606 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:34.977598    4372 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:23:36.007727 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:36.007686    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g" event={"ID":"6e08ca0d-5e22-4a83-b59e-e06c5ec74877","Type":"ContainerStarted","Data":"364d35d7b9e06276c055d23db0127074c51b3077753da6e1510656852945f6bf"}
Oct 30 19:23:36.576560 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:36.576523    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:36.577608 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:36.577580    4372 patch_prober.go:29] interesting pod/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:23:36.577772 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:36.577635    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g" podUID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 19:23:36.747572 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:36.747543    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:36.972882 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:36.972793    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"]
Oct 30 19:23:37.577480 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:37.577443    4372 patch_prober.go:29] interesting pod/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:23:37.577920 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:37.577498    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g" podUID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 19:23:38.567762 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:38.567731    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:23:38.576948 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:38.576919    4372 patch_prober.go:29] interesting pod/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:23:38.577097 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:38.576967    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g" podUID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 19:23:39.576779 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:39.576739    4372 patch_prober.go:29] interesting pod/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:23:39.577215 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:39.576802    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g" podUID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 19:23:40.407910 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:40.407875    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:40.444541 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:40.444509    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:40.576663 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:40.576623    4372 patch_prober.go:29] interesting pod/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:23:40.576851 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:40.576674    4372 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g" podUID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 19:23:41.062607 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:41.062569    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:23:41.581044 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:41.581012    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:41.581924 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:41.581905    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:23:41.974117 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:41.974031    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj" podUID="d89c0e0e-7b58-4c7f-a959-75e56c711f6f" containerName="payload-pre-processing" containerID="cri-o://4f045bda6f80c4206ddb7ebd55253bc7b05e80d74f794d12d0d4f8c3328bfaa6" gracePeriod=25
Oct 30 19:23:42.116593 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:42.116567    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:42.137643 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:42.130803    4372 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:43.077096 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:43.077063    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:23:46.676269 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:46.676229    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main" containerID="cri-o://302e2cd731be5a952c9608682fded282a4adc7344d0b69eb3e3c8898f153eb4d" gracePeriod=45
Oct 30 19:23:46.934231 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:46.934173    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 19:23:47.089839 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.089802    4372 generic.go:356] "Generic (PLEG): container finished" podID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerID="302e2cd731be5a952c9608682fded282a4adc7344d0b69eb3e3c8898f153eb4d" exitCode=0
Oct 30 19:23:47.090013 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.089850    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"
Oct 30 19:23:47.090013 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.089887    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" event={"ID":"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab","Type":"ContainerDied","Data":"302e2cd731be5a952c9608682fded282a4adc7344d0b69eb3e3c8898f153eb4d"}
Oct 30 19:23:47.090013 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.089948    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5" event={"ID":"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab","Type":"ContainerDied","Data":"a2b72d11ea497ce692013440b6af20ea723ea2129733f432d3298af810f11828"}
Oct 30 19:23:47.090013 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.089978    4372 scope.go:122] "RemoveContainer" containerID="302e2cd731be5a952c9608682fded282a4adc7344d0b69eb3e3c8898f153eb4d"
Oct 30 19:23:47.101968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.099736    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-kube-api-access-6rf5b\" (UniqueName: \"kubernetes.io/projected/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-kube-api-access-6rf5b\") pod \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") "
Oct 30 19:23:47.101968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.100225    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-dshm\") pod \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") "
Oct 30 19:23:47.101968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.100279    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-home\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-home\") pod \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") "
Oct 30 19:23:47.101968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.100329    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-tls-certs\") pod \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") "
Oct 30 19:23:47.101968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.100371    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-model-cache\") pod \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") "
Oct 30 19:23:47.101968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.100446    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-tmp-dir\") pod \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\" (UID: \"665d8ad1-52bf-4837-9a43-c21c4b2ea5ab\") "
Oct 30 19:23:47.101968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.100496    4372 scope.go:122] "RemoveContainer" containerID="302e2cd731be5a952c9608682fded282a4adc7344d0b69eb3e3c8898f153eb4d"
Oct 30 19:23:47.101968 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:23:47.101107    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"302e2cd731be5a952c9608682fded282a4adc7344d0b69eb3e3c8898f153eb4d\": container with ID starting with 302e2cd731be5a952c9608682fded282a4adc7344d0b69eb3e3c8898f153eb4d not found: ID does not exist" containerID="302e2cd731be5a952c9608682fded282a4adc7344d0b69eb3e3c8898f153eb4d"
Oct 30 19:23:47.101968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.101167    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"302e2cd731be5a952c9608682fded282a4adc7344d0b69eb3e3c8898f153eb4d"} err="failed to get container status \"302e2cd731be5a952c9608682fded282a4adc7344d0b69eb3e3c8898f153eb4d\": rpc error: code = NotFound desc = could not find container \"302e2cd731be5a952c9608682fded282a4adc7344d0b69eb3e3c8898f153eb4d\": container with ID starting with 302e2cd731be5a952c9608682fded282a4adc7344d0b69eb3e3c8898f153eb4d not found: ID does not exist"
Oct 30 19:23:47.101968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.101552    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-model-cache" pod "665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" (UID: "665d8ad1-52bf-4837-9a43-c21c4b2ea5ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:47.101968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.101947    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-home" pod "665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" (UID: "665d8ad1-52bf-4837-9a43-c21c4b2ea5ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:47.102677 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.102003    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-tmp-dir" pod "665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" (UID: "665d8ad1-52bf-4837-9a43-c21c4b2ea5ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:47.102812 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.102780    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-dshm" pod "665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" (UID: "665d8ad1-52bf-4837-9a43-c21c4b2ea5ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:47.104835 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.104799    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-tls-certs" pod "665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" (UID: "665d8ad1-52bf-4837-9a43-c21c4b2ea5ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:47.104835 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.104818    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-kube-api-access-6rf5b" pod "665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" (UID: "665d8ad1-52bf-4837-9a43-c21c4b2ea5ab"). InnerVolumeSpecName "kube-api-access-6rf5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:47.201616 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.201549    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rf5b\" (UniqueName: \"kubernetes.io/projected/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-kube-api-access-6rf5b\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:47.201616 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.201576    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:47.201616 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.201586    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:47.201616 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.201594    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:47.201616 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.201602    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:47.201883 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.201629    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:47.432944 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.432914    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"]
Oct 30 19:23:47.437673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:47.437645    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/distinct-w3-7a8fa2-kserve-7497bd4f87-wzrx5"]
Oct 30 19:23:48.265325 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:48.265293    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" path="/var/lib/kubelet/pods/665d8ad1-52bf-4837-9a43-c21c4b2ea5ab/volumes"
Oct 30 19:23:48.569335 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:48.569263    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj" podUID="d89c0e0e-7b58-4c7f-a959-75e56c711f6f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:57.695157 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:23:57.695099    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:23:58.569317 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:23:58.569269    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj" podUID="d89c0e0e-7b58-4c7f-a959-75e56c711f6f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:24:02.184325 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:02.184276    4372 generic.go:356] "Generic (PLEG): container finished" podID="d89c0e0e-7b58-4c7f-a959-75e56c711f6f" containerID="4f045bda6f80c4206ddb7ebd55253bc7b05e80d74f794d12d0d4f8c3328bfaa6" exitCode=0
Oct 30 19:24:02.184754 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:02.184346    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj" event={"ID":"d89c0e0e-7b58-4c7f-a959-75e56c711f6f","Type":"ContainerDied","Data":"4f045bda6f80c4206ddb7ebd55253bc7b05e80d74f794d12d0d4f8c3328bfaa6"}
Oct 30 19:24:02.236205 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:02.236154    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"
Oct 30 19:24:02.371876 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:02.371843    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89c0e0e-7b58-4c7f-a959-75e56c711f6f-kube-api-access-ww5rl\" (UniqueName: \"kubernetes.io/projected/d89c0e0e-7b58-4c7f-a959-75e56c711f6f-kube-api-access-ww5rl\") pod \"d89c0e0e-7b58-4c7f-a959-75e56c711f6f\" (UID: \"d89c0e0e-7b58-4c7f-a959-75e56c711f6f\") "
Oct 30 19:24:02.372070 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:02.372052    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89c0e0e-7b58-4c7f-a959-75e56c711f6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89c0e0e-7b58-4c7f-a959-75e56c711f6f-plugins-config-volume\") pod \"d89c0e0e-7b58-4c7f-a959-75e56c711f6f\" (UID: \"d89c0e0e-7b58-4c7f-a959-75e56c711f6f\") "
Oct 30 19:24:02.372515 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:02.372484    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89c0e0e-7b58-4c7f-a959-75e56c711f6f-plugins-config-volume" pod "d89c0e0e-7b58-4c7f-a959-75e56c711f6f" (UID: "d89c0e0e-7b58-4c7f-a959-75e56c711f6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:02.374332 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:02.374308    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89c0e0e-7b58-4c7f-a959-75e56c711f6f-kube-api-access-ww5rl" pod "d89c0e0e-7b58-4c7f-a959-75e56c711f6f" (UID: "d89c0e0e-7b58-4c7f-a959-75e56c711f6f"). InnerVolumeSpecName "kube-api-access-ww5rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:02.473249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:02.473225    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89c0e0e-7b58-4c7f-a959-75e56c711f6f-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:02.473249 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:02.473248    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww5rl\" (UniqueName: \"kubernetes.io/projected/d89c0e0e-7b58-4c7f-a959-75e56c711f6f-kube-api-access-ww5rl\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:03.190588 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:03.190552    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"
Oct 30 19:24:03.190588 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:03.190563    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj" event={"ID":"d89c0e0e-7b58-4c7f-a959-75e56c711f6f","Type":"ContainerDied","Data":"df299f5aba28ee387904ba4cdcfc3d5c4a6db5890449e2d1549ad3fe99b99f79"}
Oct 30 19:24:03.191192 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:03.190634    4372 scope.go:122] "RemoveContainer" containerID="4f045bda6f80c4206ddb7ebd55253bc7b05e80d74f794d12d0d4f8c3328bfaa6"
Oct 30 19:24:03.204108 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:03.204084    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:24:03.220904 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:03.220881    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"]
Oct 30 19:24:03.226745 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:03.226720    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7a8fa2-56d4699495-h6wnj"]
Oct 30 19:24:04.264940 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:04.264911    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89c0e0e-7b58-4c7f-a959-75e56c711f6f" path="/var/lib/kubelet/pods/d89c0e0e-7b58-4c7f-a959-75e56c711f6f/volumes"
Oct 30 19:24:06.698896 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:06.698859    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"]
Oct 30 19:24:06.699457 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:06.699086    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" podUID="72834257-61fb-4596-8b82-5917990f7469" containerName="istio-proxy" containerID="cri-o://52c666532f661acef0f8c6ca8f2fb7755a6d2c9edcc8259f87de867be9bb2039" gracePeriod=30
Oct 30 19:24:12.000222 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.000196    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 19:24:12.078899 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.078862    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-workload-socket\") pod \"72834257-61fb-4596-8b82-5917990f7469\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") "
Oct 30 19:24:12.079092 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.078922    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72834257-61fb-4596-8b82-5917990f7469-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72834257-61fb-4596-8b82-5917990f7469-istio-podinfo\") pod \"72834257-61fb-4596-8b82-5917990f7469\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") "
Oct 30 19:24:12.079092 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.078979    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-workload-certs\") pod \"72834257-61fb-4596-8b82-5917990f7469\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") "
Oct 30 19:24:12.079092 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.079048    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72834257-61fb-4596-8b82-5917990f7469-istio-token\" (UniqueName: \"kubernetes.io/projected/72834257-61fb-4596-8b82-5917990f7469-istio-token\") pod \"72834257-61fb-4596-8b82-5917990f7469\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") "
Oct 30 19:24:12.079092 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.079087    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-credential-socket\") pod \"72834257-61fb-4596-8b82-5917990f7469\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") "
Oct 30 19:24:12.079321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.079175    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-workload-socket" pod "72834257-61fb-4596-8b82-5917990f7469" (UID: "72834257-61fb-4596-8b82-5917990f7469"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:12.079321 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.079277    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-workload-certs" pod "72834257-61fb-4596-8b82-5917990f7469" (UID: "72834257-61fb-4596-8b82-5917990f7469"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:12.079440 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.079415    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-credential-socket" pod "72834257-61fb-4596-8b82-5917990f7469" (UID: "72834257-61fb-4596-8b82-5917990f7469"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:12.079504 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.079486    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72834257-61fb-4596-8b82-5917990f7469-kube-api-access-77c7h\" (UniqueName: \"kubernetes.io/projected/72834257-61fb-4596-8b82-5917990f7469-kube-api-access-77c7h\") pod \"72834257-61fb-4596-8b82-5917990f7469\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") "
Oct 30 19:24:12.079577 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.079551    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72834257-61fb-4596-8b82-5917990f7469-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72834257-61fb-4596-8b82-5917990f7469-istiod-ca-cert\") pod \"72834257-61fb-4596-8b82-5917990f7469\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") "
Oct 30 19:24:12.079630 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.079580    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-istio-envoy\") pod \"72834257-61fb-4596-8b82-5917990f7469\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") "
Oct 30 19:24:12.079630 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.079609    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-istio-data\") pod \"72834257-61fb-4596-8b82-5917990f7469\" (UID: \"72834257-61fb-4596-8b82-5917990f7469\") "
Oct 30 19:24:12.080247 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.080198    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.080592 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.080572    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.080738 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.080715    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.080738 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.080230    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72834257-61fb-4596-8b82-5917990f7469-istiod-ca-cert" pod "72834257-61fb-4596-8b82-5917990f7469" (UID: "72834257-61fb-4596-8b82-5917990f7469"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:12.080929 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.080538    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-istio-data" pod "72834257-61fb-4596-8b82-5917990f7469" (UID: "72834257-61fb-4596-8b82-5917990f7469"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:12.081514 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.081492    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72834257-61fb-4596-8b82-5917990f7469-istio-podinfo" pod "72834257-61fb-4596-8b82-5917990f7469" (UID: "72834257-61fb-4596-8b82-5917990f7469"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:24:12.081987 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.081968    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72834257-61fb-4596-8b82-5917990f7469-kube-api-access-77c7h" pod "72834257-61fb-4596-8b82-5917990f7469" (UID: "72834257-61fb-4596-8b82-5917990f7469"). InnerVolumeSpecName "kube-api-access-77c7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:12.082093 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.082032    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72834257-61fb-4596-8b82-5917990f7469-istio-token" pod "72834257-61fb-4596-8b82-5917990f7469" (UID: "72834257-61fb-4596-8b82-5917990f7469"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:12.082334 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.082315    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-istio-envoy" pod "72834257-61fb-4596-8b82-5917990f7469" (UID: "72834257-61fb-4596-8b82-5917990f7469"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:12.182060 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.182021    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72834257-61fb-4596-8b82-5917990f7469-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.182060 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.182062    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72834257-61fb-4596-8b82-5917990f7469-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.182270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.182080    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77c7h\" (UniqueName: \"kubernetes.io/projected/72834257-61fb-4596-8b82-5917990f7469-kube-api-access-77c7h\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.182270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.182098    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72834257-61fb-4596-8b82-5917990f7469-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.182270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.182113    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.182270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.182146    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72834257-61fb-4596-8b82-5917990f7469-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:12.246769 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.246691    4372 generic.go:356] "Generic (PLEG): container finished" podID="72834257-61fb-4596-8b82-5917990f7469" containerID="52c666532f661acef0f8c6ca8f2fb7755a6d2c9edcc8259f87de867be9bb2039" exitCode=0
Oct 30 19:24:12.246769 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.246742    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"
Oct 30 19:24:12.246967 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.246769    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" event={"ID":"72834257-61fb-4596-8b82-5917990f7469","Type":"ContainerDied","Data":"52c666532f661acef0f8c6ca8f2fb7755a6d2c9edcc8259f87de867be9bb2039"}
Oct 30 19:24:12.246967 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.246830    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94" event={"ID":"72834257-61fb-4596-8b82-5917990f7469","Type":"ContainerDied","Data":"7787357855e13b7e77bc2b02d1b17b9f45749377984dd9f371c8bd12f3b89a4c"}
Oct 30 19:24:12.246967 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.246869    4372 scope.go:122] "RemoveContainer" containerID="52c666532f661acef0f8c6ca8f2fb7755a6d2c9edcc8259f87de867be9bb2039"
Oct 30 19:24:12.257076 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.257057    4372 scope.go:122] "RemoveContainer" containerID="af5ea31094e675d08611808658b96313d4e5fcd8dd305bd3c838b87a8ba746fc"
Oct 30 19:24:12.266819 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.266799    4372 scope.go:122] "RemoveContainer" containerID="52c666532f661acef0f8c6ca8f2fb7755a6d2c9edcc8259f87de867be9bb2039"
Oct 30 19:24:12.267089 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:24:12.267072    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c666532f661acef0f8c6ca8f2fb7755a6d2c9edcc8259f87de867be9bb2039\": container with ID starting with 52c666532f661acef0f8c6ca8f2fb7755a6d2c9edcc8259f87de867be9bb2039 not found: ID does not exist" containerID="52c666532f661acef0f8c6ca8f2fb7755a6d2c9edcc8259f87de867be9bb2039"
Oct 30 19:24:12.267171 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.267096    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c666532f661acef0f8c6ca8f2fb7755a6d2c9edcc8259f87de867be9bb2039"} err="failed to get container status \"52c666532f661acef0f8c6ca8f2fb7755a6d2c9edcc8259f87de867be9bb2039\": rpc error: code = NotFound desc = could not find container \"52c666532f661acef0f8c6ca8f2fb7755a6d2c9edcc8259f87de867be9bb2039\": container with ID starting with 52c666532f661acef0f8c6ca8f2fb7755a6d2c9edcc8259f87de867be9bb2039 not found: ID does not exist"
Oct 30 19:24:12.267171 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.267111    4372 scope.go:122] "RemoveContainer" containerID="af5ea31094e675d08611808658b96313d4e5fcd8dd305bd3c838b87a8ba746fc"
Oct 30 19:24:12.267344 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:24:12.267325    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5ea31094e675d08611808658b96313d4e5fcd8dd305bd3c838b87a8ba746fc\": container with ID starting with af5ea31094e675d08611808658b96313d4e5fcd8dd305bd3c838b87a8ba746fc not found: ID does not exist" containerID="af5ea31094e675d08611808658b96313d4e5fcd8dd305bd3c838b87a8ba746fc"
Oct 30 19:24:12.267385 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.267351    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5ea31094e675d08611808658b96313d4e5fcd8dd305bd3c838b87a8ba746fc"} err="failed to get container status \"af5ea31094e675d08611808658b96313d4e5fcd8dd305bd3c838b87a8ba746fc\": rpc error: code = NotFound desc = could not find container \"af5ea31094e675d08611808658b96313d4e5fcd8dd305bd3c838b87a8ba746fc\": container with ID starting with af5ea31094e675d08611808658b96313d4e5fcd8dd305bd3c838b87a8ba746fc not found: ID does not exist"
Oct 30 19:24:12.272167 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.272114    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"]
Oct 30 19:24:12.277499 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:12.277480    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-dqx94"]
Oct 30 19:24:14.263778 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:14.263743    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72834257-61fb-4596-8b82-5917990f7469" path="/var/lib/kubelet/pods/72834257-61fb-4596-8b82-5917990f7469/volumes"
Oct 30 19:24:18.086885 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:18.086849    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:24:22.247490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:22.247459    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"]
Oct 30 19:24:22.247954 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:22.247669    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw" podUID="d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" containerName="istio-proxy" containerID="cri-o://3ff76fe5c0ca7a230ee2d8aa1ba0a3fe1b54bb65abeaa1875b80c059a3d4db6f" gracePeriod=30
Oct 30 19:24:25.635699 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:25.635661    4372 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:24:25.636114 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:25.635751    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw" podUID="d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:24:27.505462 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.505439    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 19:24:27.534977 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.534903    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-data\") pod \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") "
Oct 30 19:24:27.534977 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.534961    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-token\" (UniqueName: \"kubernetes.io/projected/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-token\") pod \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") "
Oct 30 19:24:27.535215 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.534996    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-workload-socket\") pod \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") "
Oct 30 19:24:27.535215 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.535068    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-envoy\") pod \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") "
Oct 30 19:24:27.535215 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.535097    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istiod-ca-cert\") pod \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") "
Oct 30 19:24:27.535215 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.535179    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-kube-api-access-zgmtd\" (UniqueName: \"kubernetes.io/projected/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-kube-api-access-zgmtd\") pod \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") "
Oct 30 19:24:27.535215 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.535211    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-podinfo\") pod \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") "
Oct 30 19:24:27.535491 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.535255    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-workload-certs\") pod \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") "
Oct 30 19:24:27.535491 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.535327    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-workload-socket" pod "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" (UID: "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:27.535491 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.535364    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-credential-socket\") pod \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\" (UID: \"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33\") "
Oct 30 19:24:27.535785 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.535762    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:27.536666 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.536500    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-credential-socket" pod "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" (UID: "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:27.536902 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.536878    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istiod-ca-cert" pod "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" (UID: "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:27.537207 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.537174    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-workload-certs" pod "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" (UID: "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:27.537425 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.537399    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-data" pod "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" (UID: "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:27.538092 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.538067    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-kube-api-access-zgmtd" pod "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" (UID: "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33"). InnerVolumeSpecName "kube-api-access-zgmtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:27.538703 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.538676    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-envoy" pod "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" (UID: "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:27.539356 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.539331    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-podinfo" pod "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" (UID: "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:24:27.539469 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.539432    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-token" pod "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" (UID: "d10d45d9-3b28-4c0e-aeb6-62a63f39ab33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:27.636540 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.636510    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgmtd\" (UniqueName: \"kubernetes.io/projected/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-kube-api-access-zgmtd\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:27.636540 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.636535    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:27.636676 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.636545    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:27.636676 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.636554    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:27.636676 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.636562    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:27.636676 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.636570    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:27.636676 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.636578    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:27.636676 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:27.636587    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:27.695522 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:24:27.695481    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:24:28.346664 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:28.346630    4372 generic.go:356] "Generic (PLEG): container finished" podID="d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" containerID="3ff76fe5c0ca7a230ee2d8aa1ba0a3fe1b54bb65abeaa1875b80c059a3d4db6f" exitCode=0
Oct 30 19:24:28.346808 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:28.346676    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"
Oct 30 19:24:28.346808 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:28.346696    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw" event={"ID":"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33","Type":"ContainerDied","Data":"3ff76fe5c0ca7a230ee2d8aa1ba0a3fe1b54bb65abeaa1875b80c059a3d4db6f"}
Oct 30 19:24:28.346808 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:28.346746    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw" event={"ID":"d10d45d9-3b28-4c0e-aeb6-62a63f39ab33","Type":"ContainerDied","Data":"d8fe2f3d7981b289b7ee4081183557e70180e8322839fac39bf9a9a8fc940e7d"}
Oct 30 19:24:28.346808 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:28.346775    4372 scope.go:122] "RemoveContainer" containerID="3ff76fe5c0ca7a230ee2d8aa1ba0a3fe1b54bb65abeaa1875b80c059a3d4db6f"
Oct 30 19:24:28.355949 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:28.355928    4372 scope.go:122] "RemoveContainer" containerID="3ff76fe5c0ca7a230ee2d8aa1ba0a3fe1b54bb65abeaa1875b80c059a3d4db6f"
Oct 30 19:24:28.356284 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:24:28.356262    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff76fe5c0ca7a230ee2d8aa1ba0a3fe1b54bb65abeaa1875b80c059a3d4db6f\": container with ID starting with 3ff76fe5c0ca7a230ee2d8aa1ba0a3fe1b54bb65abeaa1875b80c059a3d4db6f not found: ID does not exist" containerID="3ff76fe5c0ca7a230ee2d8aa1ba0a3fe1b54bb65abeaa1875b80c059a3d4db6f"
Oct 30 19:24:28.356352 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:28.356292    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff76fe5c0ca7a230ee2d8aa1ba0a3fe1b54bb65abeaa1875b80c059a3d4db6f"} err="failed to get container status \"3ff76fe5c0ca7a230ee2d8aa1ba0a3fe1b54bb65abeaa1875b80c059a3d4db6f\": rpc error: code = NotFound desc = could not find container \"3ff76fe5c0ca7a230ee2d8aa1ba0a3fe1b54bb65abeaa1875b80c059a3d4db6f\": container with ID starting with 3ff76fe5c0ca7a230ee2d8aa1ba0a3fe1b54bb65abeaa1875b80c059a3d4db6f not found: ID does not exist"
Oct 30 19:24:28.369673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:28.369653    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"]
Oct 30 19:24:28.371828 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:28.371805    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-6mjzw"]
Oct 30 19:24:30.264077 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:30.264039    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" path="/var/lib/kubelet/pods/d10d45d9-3b28-4c0e-aeb6-62a63f39ab33/volumes"
Oct 30 19:24:38.558306 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:24:38.558269    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:24:57.695967 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:24:57.695879    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:25:14.004983 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:14.004945    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"]
Oct 30 19:25:17.106192 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:17.106155    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:25:25.140356 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:25.140311    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"]
Oct 30 19:25:27.696062 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:25:27.696025    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:25:28.005545 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:28.005463    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:25:29.033156 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.033083    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb" podUID="ba2afdc1-190a-4aea-9cc9-2692bcaf56c7" containerName="main" containerID="cri-o://14ce2d8176b1bb3b101ce4bdfd07100c3ca3374ad45f46a93931f9d8267496f6" gracePeriod=45
Oct 30 19:25:29.049592 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.049556    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb" podUID="ba2afdc1-190a-4aea-9cc9-2692bcaf56c7" containerName="main" probeResult="failure" output="Get \"https://10.133.1.1:8000/ready\": dial tcp 10.133.1.1:8000: connect: connection refused"
Oct 30 19:25:29.280238 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.280210    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:25:29.335270 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.335239    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-tls-certs\") pod \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") "
Oct 30 19:25:29.335435 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.335285    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-dshm\") pod \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") "
Oct 30 19:25:29.335435 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.335319    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-home\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-home\") pod \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") "
Oct 30 19:25:29.335435 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.335370    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-kube-api-access-c4kpn\" (UniqueName: \"kubernetes.io/projected/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-kube-api-access-c4kpn\") pod \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") "
Oct 30 19:25:29.335435 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.335395    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-tmp-dir\") pod \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") "
Oct 30 19:25:29.335435 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.335422    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-model-cache\") pod \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\" (UID: \"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7\") "
Oct 30 19:25:29.335737 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.335625    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-home" pod "ba2afdc1-190a-4aea-9cc9-2692bcaf56c7" (UID: "ba2afdc1-190a-4aea-9cc9-2692bcaf56c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:29.335737 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.335723    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-tmp-dir" pod "ba2afdc1-190a-4aea-9cc9-2692bcaf56c7" (UID: "ba2afdc1-190a-4aea-9cc9-2692bcaf56c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:29.335909 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.335885    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-model-cache" pod "ba2afdc1-190a-4aea-9cc9-2692bcaf56c7" (UID: "ba2afdc1-190a-4aea-9cc9-2692bcaf56c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:29.335989 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.335906    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:29.335989 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.335928    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:29.337998 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.337967    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-tls-certs" pod "ba2afdc1-190a-4aea-9cc9-2692bcaf56c7" (UID: "ba2afdc1-190a-4aea-9cc9-2692bcaf56c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:29.338102 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.338057    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-dshm" pod "ba2afdc1-190a-4aea-9cc9-2692bcaf56c7" (UID: "ba2afdc1-190a-4aea-9cc9-2692bcaf56c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:29.338102 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.338062    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-kube-api-access-c4kpn" pod "ba2afdc1-190a-4aea-9cc9-2692bcaf56c7" (UID: "ba2afdc1-190a-4aea-9cc9-2692bcaf56c7"). InnerVolumeSpecName "kube-api-access-c4kpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:29.437008 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.436983    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:29.437008 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.437005    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:29.437199 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.437015    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4kpn\" (UniqueName: \"kubernetes.io/projected/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-kube-api-access-c4kpn\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:29.437199 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.437025    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:29.665664 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.665598    4372 generic.go:356] "Generic (PLEG): container finished" podID="ba2afdc1-190a-4aea-9cc9-2692bcaf56c7" containerID="14ce2d8176b1bb3b101ce4bdfd07100c3ca3374ad45f46a93931f9d8267496f6" exitCode=0
Oct 30 19:25:29.665664 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.665643    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"
Oct 30 19:25:29.665822 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.665679    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb" event={"ID":"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7","Type":"ContainerDied","Data":"14ce2d8176b1bb3b101ce4bdfd07100c3ca3374ad45f46a93931f9d8267496f6"}
Oct 30 19:25:29.665822 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.665727    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb" event={"ID":"ba2afdc1-190a-4aea-9cc9-2692bcaf56c7","Type":"ContainerDied","Data":"02d03539b2a4b35dd59884b5266237efc6b8f46a17ae9382b2f884c13fbe551a"}
Oct 30 19:25:29.665822 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.665744    4372 scope.go:122] "RemoveContainer" containerID="14ce2d8176b1bb3b101ce4bdfd07100c3ca3374ad45f46a93931f9d8267496f6"
Oct 30 19:25:29.675673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.675653    4372 scope.go:122] "RemoveContainer" containerID="14ce2d8176b1bb3b101ce4bdfd07100c3ca3374ad45f46a93931f9d8267496f6"
Oct 30 19:25:29.675963 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:25:29.675943    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14ce2d8176b1bb3b101ce4bdfd07100c3ca3374ad45f46a93931f9d8267496f6\": container with ID starting with 14ce2d8176b1bb3b101ce4bdfd07100c3ca3374ad45f46a93931f9d8267496f6 not found: ID does not exist" containerID="14ce2d8176b1bb3b101ce4bdfd07100c3ca3374ad45f46a93931f9d8267496f6"
Oct 30 19:25:29.676021 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.675973    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14ce2d8176b1bb3b101ce4bdfd07100c3ca3374ad45f46a93931f9d8267496f6"} err="failed to get container status \"14ce2d8176b1bb3b101ce4bdfd07100c3ca3374ad45f46a93931f9d8267496f6\": rpc error: code = NotFound desc = could not find container \"14ce2d8176b1bb3b101ce4bdfd07100c3ca3374ad45f46a93931f9d8267496f6\": container with ID starting with 14ce2d8176b1bb3b101ce4bdfd07100c3ca3374ad45f46a93931f9d8267496f6 not found: ID does not exist"
Oct 30 19:25:29.689462 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.689439    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"]
Oct 30 19:25:29.693904 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:29.693881    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fe8ece/test-model-fe8ece-kserve-595c7859f-b2tsb"]
Oct 30 19:25:30.265817 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:30.265774    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba2afdc1-190a-4aea-9cc9-2692bcaf56c7" path="/var/lib/kubelet/pods/ba2afdc1-190a-4aea-9cc9-2692bcaf56c7/volumes"
Oct 30 19:25:35.485624 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:35.485580    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"]
Oct 30 19:25:35.497618 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:35.497579    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"]
Oct 30 19:25:37.108689 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.108654    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"]
Oct 30 19:25:37.109173 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.108883    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn" podUID="0f86acaf-a6dd-4962-82db-f2103f9f263f" containerName="maas-api" containerID="cri-o://d0234e91e1b30dc5d6e0ae119bf141a1c0b745218d39aaf1c00f0b1e98032031" gracePeriod=30
Oct 30 19:25:37.579416 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.579382    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:25:37.709149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.709044    4372 generic.go:356] "Generic (PLEG): container finished" podID="0f86acaf-a6dd-4962-82db-f2103f9f263f" containerID="d0234e91e1b30dc5d6e0ae119bf141a1c0b745218d39aaf1c00f0b1e98032031" exitCode=0
Oct 30 19:25:37.709149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.709081    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"
Oct 30 19:25:37.709343 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.709118    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn" event={"ID":"0f86acaf-a6dd-4962-82db-f2103f9f263f","Type":"ContainerDied","Data":"d0234e91e1b30dc5d6e0ae119bf141a1c0b745218d39aaf1c00f0b1e98032031"}
Oct 30 19:25:37.709343 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.709201    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn" event={"ID":"0f86acaf-a6dd-4962-82db-f2103f9f263f","Type":"ContainerDied","Data":"921a7a4d970cdebb9b4aa7c63b1ab4995fa1ab3fd7920292ac9d7d0f55b97b89"}
Oct 30 19:25:37.709343 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.709226    4372 scope.go:122] "RemoveContainer" containerID="d0234e91e1b30dc5d6e0ae119bf141a1c0b745218d39aaf1c00f0b1e98032031"
Oct 30 19:25:37.718623 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.718595    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f86acaf-a6dd-4962-82db-f2103f9f263f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f86acaf-a6dd-4962-82db-f2103f9f263f-openshift-service-ca\") pod \"0f86acaf-a6dd-4962-82db-f2103f9f263f\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") "
Oct 30 19:25:37.718780 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.718675    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f86acaf-a6dd-4962-82db-f2103f9f263f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f86acaf-a6dd-4962-82db-f2103f9f263f-metrics-certs\") pod \"0f86acaf-a6dd-4962-82db-f2103f9f263f\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") "
Oct 30 19:25:37.718780 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.718752    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f86acaf-a6dd-4962-82db-f2103f9f263f-kube-api-access-gsxkt\" (UniqueName: \"kubernetes.io/projected/0f86acaf-a6dd-4962-82db-f2103f9f263f-kube-api-access-gsxkt\") pod \"0f86acaf-a6dd-4962-82db-f2103f9f263f\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") "
Oct 30 19:25:37.718920 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.718784    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f86acaf-a6dd-4962-82db-f2103f9f263f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f86acaf-a6dd-4962-82db-f2103f9f263f-maas-api-tls\") pod \"0f86acaf-a6dd-4962-82db-f2103f9f263f\" (UID: \"0f86acaf-a6dd-4962-82db-f2103f9f263f\") "
Oct 30 19:25:37.718996 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.718971    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f86acaf-a6dd-4962-82db-f2103f9f263f-openshift-service-ca" pod "0f86acaf-a6dd-4962-82db-f2103f9f263f" (UID: "0f86acaf-a6dd-4962-82db-f2103f9f263f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:37.719325 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.719263    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f86acaf-a6dd-4962-82db-f2103f9f263f-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:37.721418 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.721388    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f86acaf-a6dd-4962-82db-f2103f9f263f-maas-api-tls" pod "0f86acaf-a6dd-4962-82db-f2103f9f263f" (UID: "0f86acaf-a6dd-4962-82db-f2103f9f263f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:37.721529 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.721398    4372 scope.go:122] "RemoveContainer" containerID="d0234e91e1b30dc5d6e0ae119bf141a1c0b745218d39aaf1c00f0b1e98032031"
Oct 30 19:25:37.721656 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.721617    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f86acaf-a6dd-4962-82db-f2103f9f263f-metrics-certs" pod "0f86acaf-a6dd-4962-82db-f2103f9f263f" (UID: "0f86acaf-a6dd-4962-82db-f2103f9f263f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:37.721801 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:25:37.721769    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0234e91e1b30dc5d6e0ae119bf141a1c0b745218d39aaf1c00f0b1e98032031\": container with ID starting with d0234e91e1b30dc5d6e0ae119bf141a1c0b745218d39aaf1c00f0b1e98032031 not found: ID does not exist" containerID="d0234e91e1b30dc5d6e0ae119bf141a1c0b745218d39aaf1c00f0b1e98032031"
Oct 30 19:25:37.721867 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.721814    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0234e91e1b30dc5d6e0ae119bf141a1c0b745218d39aaf1c00f0b1e98032031"} err="failed to get container status \"d0234e91e1b30dc5d6e0ae119bf141a1c0b745218d39aaf1c00f0b1e98032031\": rpc error: code = NotFound desc = could not find container \"d0234e91e1b30dc5d6e0ae119bf141a1c0b745218d39aaf1c00f0b1e98032031\": container with ID starting with d0234e91e1b30dc5d6e0ae119bf141a1c0b745218d39aaf1c00f0b1e98032031 not found: ID does not exist"
Oct 30 19:25:37.721925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.721904    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f86acaf-a6dd-4962-82db-f2103f9f263f-kube-api-access-gsxkt" pod "0f86acaf-a6dd-4962-82db-f2103f9f263f" (UID: "0f86acaf-a6dd-4962-82db-f2103f9f263f"). InnerVolumeSpecName "kube-api-access-gsxkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:37.820381 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.820352    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsxkt\" (UniqueName: \"kubernetes.io/projected/0f86acaf-a6dd-4962-82db-f2103f9f263f-kube-api-access-gsxkt\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:37.820381 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.820377    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f86acaf-a6dd-4962-82db-f2103f9f263f-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:37.820511 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:37.820388    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f86acaf-a6dd-4962-82db-f2103f9f263f-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:38.035994 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:38.035968    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"]
Oct 30 19:25:38.045581 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:38.045554    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fe8ece-6d5684457c-knjnn"]
Oct 30 19:25:38.263996 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:38.263962    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f86acaf-a6dd-4962-82db-f2103f9f263f" path="/var/lib/kubelet/pods/0f86acaf-a6dd-4962-82db-f2103f9f263f/volumes"
Oct 30 19:25:40.163536 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.163495    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk" podUID="29099d2d-89eb-49da-b9ea-7e9c4914d571" containerName="main" containerID="cri-o://3aaacc29dba697399719addb7a25c2760a00dee583a22bb8166004e541a2e751" gracePeriod=45
Oct 30 19:25:40.431189 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.431163    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:25:40.486914 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.486873    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m" podUID="211df401-6e2e-4db2-a0c9-4e0d24532573" containerName="payload-processing" containerID="cri-o://56c752be625aa21e550494c7f32f4910c42705260b4a75fa1c45fb44bcbdacbe" gracePeriod=25
Oct 30 19:25:40.498355 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.498326    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx" podUID="33dfa4ab-12fb-4444-b419-9c24113ea33e" containerName="payload-pre-processing" containerID="cri-o://bd024ffbf4adf61812be97fc923d6391779fef790151d3cd3ffe3d1925aecae8" gracePeriod=25
Oct 30 19:25:40.549728 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.549706    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-model-cache\") pod \"29099d2d-89eb-49da-b9ea-7e9c4914d571\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") "
Oct 30 19:25:40.549853 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.549774    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29099d2d-89eb-49da-b9ea-7e9c4914d571-tls-certs\" (UniqueName: \"kubernetes.io/secret/29099d2d-89eb-49da-b9ea-7e9c4914d571-tls-certs\") pod \"29099d2d-89eb-49da-b9ea-7e9c4914d571\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") "
Oct 30 19:25:40.549853 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.549833    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-home\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-home\") pod \"29099d2d-89eb-49da-b9ea-7e9c4914d571\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") "
Oct 30 19:25:40.549998 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.549880    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-tmp-dir\") pod \"29099d2d-89eb-49da-b9ea-7e9c4914d571\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") "
Oct 30 19:25:40.549998 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.549934    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-dshm\") pod \"29099d2d-89eb-49da-b9ea-7e9c4914d571\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") "
Oct 30 19:25:40.550115 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.549993    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-model-cache" pod "29099d2d-89eb-49da-b9ea-7e9c4914d571" (UID: "29099d2d-89eb-49da-b9ea-7e9c4914d571"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:40.550115 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.549999    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29099d2d-89eb-49da-b9ea-7e9c4914d571-kube-api-access-hgfmk\" (UniqueName: \"kubernetes.io/projected/29099d2d-89eb-49da-b9ea-7e9c4914d571-kube-api-access-hgfmk\") pod \"29099d2d-89eb-49da-b9ea-7e9c4914d571\" (UID: \"29099d2d-89eb-49da-b9ea-7e9c4914d571\") "
Oct 30 19:25:40.550268 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.550114    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-home" pod "29099d2d-89eb-49da-b9ea-7e9c4914d571" (UID: "29099d2d-89eb-49da-b9ea-7e9c4914d571"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:40.550268 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.550167    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-tmp-dir" pod "29099d2d-89eb-49da-b9ea-7e9c4914d571" (UID: "29099d2d-89eb-49da-b9ea-7e9c4914d571"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:40.550470 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.550451    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:40.550545 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.550475    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:40.550545 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.550489    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:40.552120 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.552098    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29099d2d-89eb-49da-b9ea-7e9c4914d571-kube-api-access-hgfmk" pod "29099d2d-89eb-49da-b9ea-7e9c4914d571" (UID: "29099d2d-89eb-49da-b9ea-7e9c4914d571"). InnerVolumeSpecName "kube-api-access-hgfmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:40.552239 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.552190    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29099d2d-89eb-49da-b9ea-7e9c4914d571-tls-certs" pod "29099d2d-89eb-49da-b9ea-7e9c4914d571" (UID: "29099d2d-89eb-49da-b9ea-7e9c4914d571"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:40.552279 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.552229    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-dshm" pod "29099d2d-89eb-49da-b9ea-7e9c4914d571" (UID: "29099d2d-89eb-49da-b9ea-7e9c4914d571"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:40.651454 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.651428    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29099d2d-89eb-49da-b9ea-7e9c4914d571-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:40.651577 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.651457    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgfmk\" (UniqueName: \"kubernetes.io/projected/29099d2d-89eb-49da-b9ea-7e9c4914d571-kube-api-access-hgfmk\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:40.651577 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.651467    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29099d2d-89eb-49da-b9ea-7e9c4914d571-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:40.711697 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.711671    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"
Oct 30 19:25:40.726659 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.726631    4372 generic.go:356] "Generic (PLEG): container finished" podID="29099d2d-89eb-49da-b9ea-7e9c4914d571" containerID="3aaacc29dba697399719addb7a25c2760a00dee583a22bb8166004e541a2e751" exitCode=0
Oct 30 19:25:40.726794 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.726722    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk" event={"ID":"29099d2d-89eb-49da-b9ea-7e9c4914d571","Type":"ContainerDied","Data":"3aaacc29dba697399719addb7a25c2760a00dee583a22bb8166004e541a2e751"}
Oct 30 19:25:40.726794 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.726772    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk" event={"ID":"29099d2d-89eb-49da-b9ea-7e9c4914d571","Type":"ContainerDied","Data":"9bfcf0d5035d931e62d8566132be572702af19d34b1b17258264ad42e8d10bf3"}
Oct 30 19:25:40.726794 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.726793    4372 scope.go:122] "RemoveContainer" containerID="3aaacc29dba697399719addb7a25c2760a00dee583a22bb8166004e541a2e751"
Oct 30 19:25:40.726927 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.726736    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"
Oct 30 19:25:40.729302 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.729281    4372 generic.go:356] "Generic (PLEG): container finished" podID="211df401-6e2e-4db2-a0c9-4e0d24532573" containerID="56c752be625aa21e550494c7f32f4910c42705260b4a75fa1c45fb44bcbdacbe" exitCode=0
Oct 30 19:25:40.729393 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.729319    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"
Oct 30 19:25:40.729393 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.729349    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m" event={"ID":"211df401-6e2e-4db2-a0c9-4e0d24532573","Type":"ContainerDied","Data":"56c752be625aa21e550494c7f32f4910c42705260b4a75fa1c45fb44bcbdacbe"}
Oct 30 19:25:40.729393 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.729388    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m" event={"ID":"211df401-6e2e-4db2-a0c9-4e0d24532573","Type":"ContainerDied","Data":"c2f076b87114dac8491c2df5d30067242af57ca53361cefe26916734e15aa902"}
Oct 30 19:25:40.738674 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.738651    4372 scope.go:122] "RemoveContainer" containerID="3aaacc29dba697399719addb7a25c2760a00dee583a22bb8166004e541a2e751"
Oct 30 19:25:40.739057 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:25:40.739028    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aaacc29dba697399719addb7a25c2760a00dee583a22bb8166004e541a2e751\": container with ID starting with 3aaacc29dba697399719addb7a25c2760a00dee583a22bb8166004e541a2e751 not found: ID does not exist" containerID="3aaacc29dba697399719addb7a25c2760a00dee583a22bb8166004e541a2e751"
Oct 30 19:25:40.739491 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.739066    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aaacc29dba697399719addb7a25c2760a00dee583a22bb8166004e541a2e751"} err="failed to get container status \"3aaacc29dba697399719addb7a25c2760a00dee583a22bb8166004e541a2e751\": rpc error: code = NotFound desc = could not find container \"3aaacc29dba697399719addb7a25c2760a00dee583a22bb8166004e541a2e751\": container with ID starting with 3aaacc29dba697399719addb7a25c2760a00dee583a22bb8166004e541a2e751 not found: ID does not exist"
Oct 30 19:25:40.739491 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.739090    4372 scope.go:122] "RemoveContainer" containerID="56c752be625aa21e550494c7f32f4910c42705260b4a75fa1c45fb44bcbdacbe"
Oct 30 19:25:40.763319 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.763277    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:25:40.768394 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.768057    4372 scope.go:122] "RemoveContainer" containerID="56c752be625aa21e550494c7f32f4910c42705260b4a75fa1c45fb44bcbdacbe"
Oct 30 19:25:40.769565 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:25:40.768786    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c752be625aa21e550494c7f32f4910c42705260b4a75fa1c45fb44bcbdacbe\": container with ID starting with 56c752be625aa21e550494c7f32f4910c42705260b4a75fa1c45fb44bcbdacbe not found: ID does not exist" containerID="56c752be625aa21e550494c7f32f4910c42705260b4a75fa1c45fb44bcbdacbe"
Oct 30 19:25:40.769565 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.768825    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c752be625aa21e550494c7f32f4910c42705260b4a75fa1c45fb44bcbdacbe"} err="failed to get container status \"56c752be625aa21e550494c7f32f4910c42705260b4a75fa1c45fb44bcbdacbe\": rpc error: code = NotFound desc = could not find container \"56c752be625aa21e550494c7f32f4910c42705260b4a75fa1c45fb44bcbdacbe\": container with ID starting with 56c752be625aa21e550494c7f32f4910c42705260b4a75fa1c45fb44bcbdacbe not found: ID does not exist"
Oct 30 19:25:40.792297 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.792258    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"]
Oct 30 19:25:40.795319 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.795290    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-58bb38/test-model-58bb38-kserve-7c7bf94f5f-vx5tk"]
Oct 30 19:25:40.855387 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.855359    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/211df401-6e2e-4db2-a0c9-4e0d24532573-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211df401-6e2e-4db2-a0c9-4e0d24532573-plugins-config-volume\") pod \"211df401-6e2e-4db2-a0c9-4e0d24532573\" (UID: \"211df401-6e2e-4db2-a0c9-4e0d24532573\") "
Oct 30 19:25:40.855566 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.855468    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/211df401-6e2e-4db2-a0c9-4e0d24532573-kube-api-access-8qspm\" (UniqueName: \"kubernetes.io/projected/211df401-6e2e-4db2-a0c9-4e0d24532573-kube-api-access-8qspm\") pod \"211df401-6e2e-4db2-a0c9-4e0d24532573\" (UID: \"211df401-6e2e-4db2-a0c9-4e0d24532573\") "
Oct 30 19:25:40.855774 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.855746    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211df401-6e2e-4db2-a0c9-4e0d24532573-plugins-config-volume" pod "211df401-6e2e-4db2-a0c9-4e0d24532573" (UID: "211df401-6e2e-4db2-a0c9-4e0d24532573"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:40.857875 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.857844    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211df401-6e2e-4db2-a0c9-4e0d24532573-kube-api-access-8qspm" pod "211df401-6e2e-4db2-a0c9-4e0d24532573" (UID: "211df401-6e2e-4db2-a0c9-4e0d24532573"). InnerVolumeSpecName "kube-api-access-8qspm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:40.956486 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.956432    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qspm\" (UniqueName: \"kubernetes.io/projected/211df401-6e2e-4db2-a0c9-4e0d24532573-kube-api-access-8qspm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:40.956486 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:40.956455    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211df401-6e2e-4db2-a0c9-4e0d24532573-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:41.053523 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:41.053494    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"]
Oct 30 19:25:41.060926 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:41.060903    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fe8ece-f6c5648d-zbk8m"]
Oct 30 19:25:42.265335 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:42.265306    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="211df401-6e2e-4db2-a0c9-4e0d24532573" path="/var/lib/kubelet/pods/211df401-6e2e-4db2-a0c9-4e0d24532573/volumes"
Oct 30 19:25:42.265749 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:42.265698    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29099d2d-89eb-49da-b9ea-7e9c4914d571" path="/var/lib/kubelet/pods/29099d2d-89eb-49da-b9ea-7e9c4914d571/volumes"
Oct 30 19:25:46.008640 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:46.008606    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"]
Oct 30 19:25:46.009097 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:46.008791    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v" podUID="450a9e2f-36a4-419e-a586-981cda482d97" containerName="istio-proxy" containerID="cri-o://f5ffdb0cb85780d7db97c2a1c2bf4a6ca2b8e75afb082ae7338800c0dd4d7b5f" gracePeriod=30
Oct 30 19:25:46.462074 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:46.462032    4372 patch_prober.go:29] interesting pod/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:25:46.462276 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:46.462089    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v" podUID="450a9e2f-36a4-419e-a586-981cda482d97" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:25:46.576920 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:46.576876    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx" podUID="33dfa4ab-12fb-4444-b419-9c24113ea33e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:25:51.269681 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.269656    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:25:51.460330 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.460301    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450a9e2f-36a4-419e-a586-981cda482d97-kube-api-access-nfffs\" (UniqueName: \"kubernetes.io/projected/450a9e2f-36a4-419e-a586-981cda482d97-kube-api-access-nfffs\") pod \"450a9e2f-36a4-419e-a586-981cda482d97\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") "
Oct 30 19:25:51.460514 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.460344    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-istio-envoy\") pod \"450a9e2f-36a4-419e-a586-981cda482d97\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") "
Oct 30 19:25:51.460514 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.460488    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-workload-certs\") pod \"450a9e2f-36a4-419e-a586-981cda482d97\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") "
Oct 30 19:25:51.460664 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.460548    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-istio-data\") pod \"450a9e2f-36a4-419e-a586-981cda482d97\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") "
Oct 30 19:25:51.460664 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.460585    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/450a9e2f-36a4-419e-a586-981cda482d97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/450a9e2f-36a4-419e-a586-981cda482d97-istio-podinfo\") pod \"450a9e2f-36a4-419e-a586-981cda482d97\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") "
Oct 30 19:25:51.460784 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.460662    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450a9e2f-36a4-419e-a586-981cda482d97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/450a9e2f-36a4-419e-a586-981cda482d97-istiod-ca-cert\") pod \"450a9e2f-36a4-419e-a586-981cda482d97\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") "
Oct 30 19:25:51.460784 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.460714    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-credential-socket\") pod \"450a9e2f-36a4-419e-a586-981cda482d97\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") "
Oct 30 19:25:51.460784 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.460745    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450a9e2f-36a4-419e-a586-981cda482d97-istio-token\" (UniqueName: \"kubernetes.io/projected/450a9e2f-36a4-419e-a586-981cda482d97-istio-token\") pod \"450a9e2f-36a4-419e-a586-981cda482d97\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") "
Oct 30 19:25:51.460784 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.460778    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-workload-socket\") pod \"450a9e2f-36a4-419e-a586-981cda482d97\" (UID: \"450a9e2f-36a4-419e-a586-981cda482d97\") "
Oct 30 19:25:51.461116 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.460750    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-workload-certs" pod "450a9e2f-36a4-419e-a586-981cda482d97" (UID: "450a9e2f-36a4-419e-a586-981cda482d97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:51.461234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.460815    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-istio-data" pod "450a9e2f-36a4-419e-a586-981cda482d97" (UID: "450a9e2f-36a4-419e-a586-981cda482d97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:51.461234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.460990    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-credential-socket" pod "450a9e2f-36a4-419e-a586-981cda482d97" (UID: "450a9e2f-36a4-419e-a586-981cda482d97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:51.461234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.461031    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450a9e2f-36a4-419e-a586-981cda482d97-istiod-ca-cert" pod "450a9e2f-36a4-419e-a586-981cda482d97" (UID: "450a9e2f-36a4-419e-a586-981cda482d97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:51.461417 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.461393    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-workload-socket" pod "450a9e2f-36a4-419e-a586-981cda482d97" (UID: "450a9e2f-36a4-419e-a586-981cda482d97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:51.462836 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.462798    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450a9e2f-36a4-419e-a586-981cda482d97-kube-api-access-nfffs" pod "450a9e2f-36a4-419e-a586-981cda482d97" (UID: "450a9e2f-36a4-419e-a586-981cda482d97"). InnerVolumeSpecName "kube-api-access-nfffs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:51.463086 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.463069    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450a9e2f-36a4-419e-a586-981cda482d97-istio-token" pod "450a9e2f-36a4-419e-a586-981cda482d97" (UID: "450a9e2f-36a4-419e-a586-981cda482d97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:51.463256 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.463236    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-istio-envoy" pod "450a9e2f-36a4-419e-a586-981cda482d97" (UID: "450a9e2f-36a4-419e-a586-981cda482d97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:51.463348 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.463252    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/450a9e2f-36a4-419e-a586-981cda482d97-istio-podinfo" pod "450a9e2f-36a4-419e-a586-981cda482d97" (UID: "450a9e2f-36a4-419e-a586-981cda482d97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:25:51.561813 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.561785    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:51.561813 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.561813    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/450a9e2f-36a4-419e-a586-981cda482d97-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:51.561968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.561828    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/450a9e2f-36a4-419e-a586-981cda482d97-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:51.561968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.561841    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:51.561968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.561853    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/450a9e2f-36a4-419e-a586-981cda482d97-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:51.561968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.561867    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:51.561968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.561881    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfffs\" (UniqueName: \"kubernetes.io/projected/450a9e2f-36a4-419e-a586-981cda482d97-kube-api-access-nfffs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:51.561968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.561896    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:51.561968 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.561909    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/450a9e2f-36a4-419e-a586-981cda482d97-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:51.800688 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.800626    4372 generic.go:356] "Generic (PLEG): container finished" podID="450a9e2f-36a4-419e-a586-981cda482d97" containerID="f5ffdb0cb85780d7db97c2a1c2bf4a6ca2b8e75afb082ae7338800c0dd4d7b5f" exitCode=0
Oct 30 19:25:51.800688 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.800675    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"
Oct 30 19:25:51.800855 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.800716    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v" event={"ID":"450a9e2f-36a4-419e-a586-981cda482d97","Type":"ContainerDied","Data":"f5ffdb0cb85780d7db97c2a1c2bf4a6ca2b8e75afb082ae7338800c0dd4d7b5f"}
Oct 30 19:25:51.800855 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.800775    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v" event={"ID":"450a9e2f-36a4-419e-a586-981cda482d97","Type":"ContainerDied","Data":"e08d1f08c830412073ddb446c30348a3e1368071fef3db16807d3f0e38126e54"}
Oct 30 19:25:51.800855 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.800805    4372 scope.go:122] "RemoveContainer" containerID="f5ffdb0cb85780d7db97c2a1c2bf4a6ca2b8e75afb082ae7338800c0dd4d7b5f"
Oct 30 19:25:51.810341 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.810323    4372 scope.go:122] "RemoveContainer" containerID="a733c275f1c2935a125d0ebcd73118214a3d414c342f9db476ea702321d5e396"
Oct 30 19:25:51.818603 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.818585    4372 scope.go:122] "RemoveContainer" containerID="f5ffdb0cb85780d7db97c2a1c2bf4a6ca2b8e75afb082ae7338800c0dd4d7b5f"
Oct 30 19:25:51.818906 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:25:51.818888    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ffdb0cb85780d7db97c2a1c2bf4a6ca2b8e75afb082ae7338800c0dd4d7b5f\": container with ID starting with f5ffdb0cb85780d7db97c2a1c2bf4a6ca2b8e75afb082ae7338800c0dd4d7b5f not found: ID does not exist" containerID="f5ffdb0cb85780d7db97c2a1c2bf4a6ca2b8e75afb082ae7338800c0dd4d7b5f"
Oct 30 19:25:51.818962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.818914    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ffdb0cb85780d7db97c2a1c2bf4a6ca2b8e75afb082ae7338800c0dd4d7b5f"} err="failed to get container status \"f5ffdb0cb85780d7db97c2a1c2bf4a6ca2b8e75afb082ae7338800c0dd4d7b5f\": rpc error: code = NotFound desc = could not find container \"f5ffdb0cb85780d7db97c2a1c2bf4a6ca2b8e75afb082ae7338800c0dd4d7b5f\": container with ID starting with f5ffdb0cb85780d7db97c2a1c2bf4a6ca2b8e75afb082ae7338800c0dd4d7b5f not found: ID does not exist"
Oct 30 19:25:51.818962 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.818930    4372 scope.go:122] "RemoveContainer" containerID="a733c275f1c2935a125d0ebcd73118214a3d414c342f9db476ea702321d5e396"
Oct 30 19:25:51.819214 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:25:51.819189    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a733c275f1c2935a125d0ebcd73118214a3d414c342f9db476ea702321d5e396\": container with ID starting with a733c275f1c2935a125d0ebcd73118214a3d414c342f9db476ea702321d5e396 not found: ID does not exist" containerID="a733c275f1c2935a125d0ebcd73118214a3d414c342f9db476ea702321d5e396"
Oct 30 19:25:51.819316 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.819222    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a733c275f1c2935a125d0ebcd73118214a3d414c342f9db476ea702321d5e396"} err="failed to get container status \"a733c275f1c2935a125d0ebcd73118214a3d414c342f9db476ea702321d5e396\": rpc error: code = NotFound desc = could not find container \"a733c275f1c2935a125d0ebcd73118214a3d414c342f9db476ea702321d5e396\": container with ID starting with a733c275f1c2935a125d0ebcd73118214a3d414c342f9db476ea702321d5e396 not found: ID does not exist"
Oct 30 19:25:51.826250 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.826226    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"]
Oct 30 19:25:51.831199 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:51.831176    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fe8ece-openshift-default-7f7d45cdc-hn25v"]
Oct 30 19:25:52.264009 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:52.263974    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450a9e2f-36a4-419e-a586-981cda482d97" path="/var/lib/kubelet/pods/450a9e2f-36a4-419e-a586-981cda482d97/volumes"
Oct 30 19:25:56.576990 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:56.576951    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx" podUID="33dfa4ab-12fb-4444-b419-9c24113ea33e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:25:56.674184 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:56.674145    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"]
Oct 30 19:25:56.691792 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:25:56.691758    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"]
Oct 30 19:25:57.697154 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:25:57.697098    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:26:00.118068 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.118033    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"]
Oct 30 19:26:00.118553 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.118249    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r" podUID="8144c849-ae43-4536-b9d1-4dbdde207847" containerName="maas-api" containerID="cri-o://c1b29206cd9dbc4e15056d9a195fda2e1b5eb117c06d4f6154d3044623f466fa" gracePeriod=30
Oct 30 19:26:00.756429 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.756366    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"
Oct 30 19:26:00.840892 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.840868    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33dfa4ab-12fb-4444-b419-9c24113ea33e-kube-api-access-5vlhs\" (UniqueName: \"kubernetes.io/projected/33dfa4ab-12fb-4444-b419-9c24113ea33e-kube-api-access-5vlhs\") pod \"33dfa4ab-12fb-4444-b419-9c24113ea33e\" (UID: \"33dfa4ab-12fb-4444-b419-9c24113ea33e\") "
Oct 30 19:26:00.841046 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.840909    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33dfa4ab-12fb-4444-b419-9c24113ea33e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33dfa4ab-12fb-4444-b419-9c24113ea33e-plugins-config-volume\") pod \"33dfa4ab-12fb-4444-b419-9c24113ea33e\" (UID: \"33dfa4ab-12fb-4444-b419-9c24113ea33e\") "
Oct 30 19:26:00.841352 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.841327    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33dfa4ab-12fb-4444-b419-9c24113ea33e-plugins-config-volume" pod "33dfa4ab-12fb-4444-b419-9c24113ea33e" (UID: "33dfa4ab-12fb-4444-b419-9c24113ea33e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:00.843165 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.843141    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33dfa4ab-12fb-4444-b419-9c24113ea33e-kube-api-access-5vlhs" pod "33dfa4ab-12fb-4444-b419-9c24113ea33e" (UID: "33dfa4ab-12fb-4444-b419-9c24113ea33e"). InnerVolumeSpecName "kube-api-access-5vlhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:00.850926 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.850898    4372 generic.go:356] "Generic (PLEG): container finished" podID="33dfa4ab-12fb-4444-b419-9c24113ea33e" containerID="bd024ffbf4adf61812be97fc923d6391779fef790151d3cd3ffe3d1925aecae8" exitCode=0
Oct 30 19:26:00.851045 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.850937    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"
Oct 30 19:26:00.851045 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.850979    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx" event={"ID":"33dfa4ab-12fb-4444-b419-9c24113ea33e","Type":"ContainerDied","Data":"bd024ffbf4adf61812be97fc923d6391779fef790151d3cd3ffe3d1925aecae8"}
Oct 30 19:26:00.851045 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.851041    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx" event={"ID":"33dfa4ab-12fb-4444-b419-9c24113ea33e","Type":"ContainerDied","Data":"16fd16d632cbc386fbe94ac1d90ebb3800627f819c06b979d32f523799821cbb"}
Oct 30 19:26:00.851246 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.851071    4372 scope.go:122] "RemoveContainer" containerID="bd024ffbf4adf61812be97fc923d6391779fef790151d3cd3ffe3d1925aecae8"
Oct 30 19:26:00.860004 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.859988    4372 scope.go:122] "RemoveContainer" containerID="bd024ffbf4adf61812be97fc923d6391779fef790151d3cd3ffe3d1925aecae8"
Oct 30 19:26:00.860305 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:26:00.860286    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd024ffbf4adf61812be97fc923d6391779fef790151d3cd3ffe3d1925aecae8\": container with ID starting with bd024ffbf4adf61812be97fc923d6391779fef790151d3cd3ffe3d1925aecae8 not found: ID does not exist" containerID="bd024ffbf4adf61812be97fc923d6391779fef790151d3cd3ffe3d1925aecae8"
Oct 30 19:26:00.860363 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.860313    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd024ffbf4adf61812be97fc923d6391779fef790151d3cd3ffe3d1925aecae8"} err="failed to get container status \"bd024ffbf4adf61812be97fc923d6391779fef790151d3cd3ffe3d1925aecae8\": rpc error: code = NotFound desc = could not find container \"bd024ffbf4adf61812be97fc923d6391779fef790151d3cd3ffe3d1925aecae8\": container with ID starting with bd024ffbf4adf61812be97fc923d6391779fef790151d3cd3ffe3d1925aecae8 not found: ID does not exist"
Oct 30 19:26:00.874678 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.874657    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"]
Oct 30 19:26:00.878961 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.878940    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fe8ece-86688cb8c5-9skzx"]
Oct 30 19:26:00.942262 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.942239    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vlhs\" (UniqueName: \"kubernetes.io/projected/33dfa4ab-12fb-4444-b419-9c24113ea33e-kube-api-access-5vlhs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:00.942361 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:00.942264    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33dfa4ab-12fb-4444-b419-9c24113ea33e-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:01.361431 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.361403    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:26:01.447452 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.447386    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8144c849-ae43-4536-b9d1-4dbdde207847-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8144c849-ae43-4536-b9d1-4dbdde207847-maas-api-tls\") pod \"8144c849-ae43-4536-b9d1-4dbdde207847\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") "
Oct 30 19:26:01.447634 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.447482    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8144c849-ae43-4536-b9d1-4dbdde207847-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8144c849-ae43-4536-b9d1-4dbdde207847-metrics-certs\") pod \"8144c849-ae43-4536-b9d1-4dbdde207847\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") "
Oct 30 19:26:01.447634 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.447579    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8144c849-ae43-4536-b9d1-4dbdde207847-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8144c849-ae43-4536-b9d1-4dbdde207847-openshift-service-ca\") pod \"8144c849-ae43-4536-b9d1-4dbdde207847\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") "
Oct 30 19:26:01.447717 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.447642    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8144c849-ae43-4536-b9d1-4dbdde207847-kube-api-access-79cqf\" (UniqueName: \"kubernetes.io/projected/8144c849-ae43-4536-b9d1-4dbdde207847-kube-api-access-79cqf\") pod \"8144c849-ae43-4536-b9d1-4dbdde207847\" (UID: \"8144c849-ae43-4536-b9d1-4dbdde207847\") "
Oct 30 19:26:01.447964 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.447932    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8144c849-ae43-4536-b9d1-4dbdde207847-openshift-service-ca" pod "8144c849-ae43-4536-b9d1-4dbdde207847" (UID: "8144c849-ae43-4536-b9d1-4dbdde207847"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:01.449755 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.449730    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8144c849-ae43-4536-b9d1-4dbdde207847-maas-api-tls" pod "8144c849-ae43-4536-b9d1-4dbdde207847" (UID: "8144c849-ae43-4536-b9d1-4dbdde207847"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:01.449866 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.449777    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8144c849-ae43-4536-b9d1-4dbdde207847-metrics-certs" pod "8144c849-ae43-4536-b9d1-4dbdde207847" (UID: "8144c849-ae43-4536-b9d1-4dbdde207847"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:01.449866 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.449802    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8144c849-ae43-4536-b9d1-4dbdde207847-kube-api-access-79cqf" pod "8144c849-ae43-4536-b9d1-4dbdde207847" (UID: "8144c849-ae43-4536-b9d1-4dbdde207847"). InnerVolumeSpecName "kube-api-access-79cqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:01.548674 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.548631    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8144c849-ae43-4536-b9d1-4dbdde207847-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:01.548674 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.548669    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8144c849-ae43-4536-b9d1-4dbdde207847-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:01.548860 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.548687    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8144c849-ae43-4536-b9d1-4dbdde207847-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:01.548860 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.548700    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79cqf\" (UniqueName: \"kubernetes.io/projected/8144c849-ae43-4536-b9d1-4dbdde207847-kube-api-access-79cqf\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:01.675242 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.675177    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn" podUID="34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665" containerName="payload-processing" containerID="cri-o://63378dd8fdbfd98bdc25df977fc8591ab54c3cc5d25962d19bce8c9887ef12c6" gracePeriod=25
Oct 30 19:26:01.692655 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.692625    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m" podUID="d49ae273-a4bf-4aae-bbac-6ed7c086f024" containerName="payload-pre-processing" containerID="cri-o://42b1561c2e5e2224c93a3f530c77cecc6d00bf7915d7694111a89c8117aec5a2" gracePeriod=25
Oct 30 19:26:01.861371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.861330    4372 generic.go:356] "Generic (PLEG): container finished" podID="34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665" containerID="63378dd8fdbfd98bdc25df977fc8591ab54c3cc5d25962d19bce8c9887ef12c6" exitCode=0
Oct 30 19:26:01.861528 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.861385    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn" event={"ID":"34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665","Type":"ContainerDied","Data":"63378dd8fdbfd98bdc25df977fc8591ab54c3cc5d25962d19bce8c9887ef12c6"}
Oct 30 19:26:01.863403 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.863374    4372 generic.go:356] "Generic (PLEG): container finished" podID="8144c849-ae43-4536-b9d1-4dbdde207847" containerID="c1b29206cd9dbc4e15056d9a195fda2e1b5eb117c06d4f6154d3044623f466fa" exitCode=0
Oct 30 19:26:01.863564 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.863408    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r" event={"ID":"8144c849-ae43-4536-b9d1-4dbdde207847","Type":"ContainerDied","Data":"c1b29206cd9dbc4e15056d9a195fda2e1b5eb117c06d4f6154d3044623f466fa"}
Oct 30 19:26:01.863564 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.863438    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"
Oct 30 19:26:01.863564 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.863467    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r" event={"ID":"8144c849-ae43-4536-b9d1-4dbdde207847","Type":"ContainerDied","Data":"b2e7b500a5e2b86ec56d623a958411451d034022cf219f43ef19ec2212842b87"}
Oct 30 19:26:01.863564 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.863501    4372 scope.go:122] "RemoveContainer" containerID="c1b29206cd9dbc4e15056d9a195fda2e1b5eb117c06d4f6154d3044623f466fa"
Oct 30 19:26:01.875235 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.875213    4372 scope.go:122] "RemoveContainer" containerID="c1b29206cd9dbc4e15056d9a195fda2e1b5eb117c06d4f6154d3044623f466fa"
Oct 30 19:26:01.875510 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:26:01.875486    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b29206cd9dbc4e15056d9a195fda2e1b5eb117c06d4f6154d3044623f466fa\": container with ID starting with c1b29206cd9dbc4e15056d9a195fda2e1b5eb117c06d4f6154d3044623f466fa not found: ID does not exist" containerID="c1b29206cd9dbc4e15056d9a195fda2e1b5eb117c06d4f6154d3044623f466fa"
Oct 30 19:26:01.875600 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.875519    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b29206cd9dbc4e15056d9a195fda2e1b5eb117c06d4f6154d3044623f466fa"} err="failed to get container status \"c1b29206cd9dbc4e15056d9a195fda2e1b5eb117c06d4f6154d3044623f466fa\": rpc error: code = NotFound desc = could not find container \"c1b29206cd9dbc4e15056d9a195fda2e1b5eb117c06d4f6154d3044623f466fa\": container with ID starting with c1b29206cd9dbc4e15056d9a195fda2e1b5eb117c06d4f6154d3044623f466fa not found: ID does not exist"
Oct 30 19:26:01.895036 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.895010    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"]
Oct 30 19:26:01.898893 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.898871    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58bb38-769587d694-hwg7r"]
Oct 30 19:26:01.931951 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.931926    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"
Oct 30 19:26:01.954604 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.954585    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665-kube-api-access-6b928\" (UniqueName: \"kubernetes.io/projected/34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665-kube-api-access-6b928\") pod \"34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665\" (UID: \"34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665\") "
Oct 30 19:26:01.954701 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.954626    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665-plugins-config-volume\") pod \"34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665\" (UID: \"34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665\") "
Oct 30 19:26:01.955112 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.955075    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665-plugins-config-volume" pod "34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665" (UID: "34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:01.957268 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:01.957242    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665-kube-api-access-6b928" pod "34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665" (UID: "34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665"). InnerVolumeSpecName "kube-api-access-6b928". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:02.056277 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:02.056220    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b928\" (UniqueName: \"kubernetes.io/projected/34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665-kube-api-access-6b928\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:02.056277 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:02.056243    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:02.263697 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:02.263661    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33dfa4ab-12fb-4444-b419-9c24113ea33e" path="/var/lib/kubelet/pods/33dfa4ab-12fb-4444-b419-9c24113ea33e/volumes"
Oct 30 19:26:02.264157 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:02.264110    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8144c849-ae43-4536-b9d1-4dbdde207847" path="/var/lib/kubelet/pods/8144c849-ae43-4536-b9d1-4dbdde207847/volumes"
Oct 30 19:26:02.815664 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:02.815630    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m" podUID="d49ae273-a4bf-4aae-bbac-6ed7c086f024" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:26:02.871582 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:02.871556    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"
Oct 30 19:26:02.871746 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:02.871595    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn" event={"ID":"34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665","Type":"ContainerDied","Data":"7e59235f1b660358a560aa0f05aa1897680f195f46ad04a03d62df4fcceabd8c"}
Oct 30 19:26:02.871746 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:02.871662    4372 scope.go:122] "RemoveContainer" containerID="63378dd8fdbfd98bdc25df977fc8591ab54c3cc5d25962d19bce8c9887ef12c6"
Oct 30 19:26:02.902261 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:02.902235    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"]
Oct 30 19:26:02.904274 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:02.904248    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58bb38-c7c7db59d-zgtkn"]
Oct 30 19:26:03.472856 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:03.472823    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:26:04.264103 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:04.264068    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665" path="/var/lib/kubelet/pods/34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665/volumes"
Oct 30 19:26:09.468020 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:09.467982    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"]
Oct 30 19:26:09.468516 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:09.468284    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g" podUID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" containerName="istio-proxy" containerID="cri-o://364d35d7b9e06276c055d23db0127074c51b3077753da6e1510656852945f6bf" gracePeriod=30
Oct 30 19:26:11.583965 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:11.583926    4372 patch_prober.go:29] interesting pod/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:26:11.584516 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:11.583978    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g" podUID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:26:12.815619 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:12.815581    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m" podUID="d49ae273-a4bf-4aae-bbac-6ed7c086f024" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:26:14.725176 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.725139    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:26:14.773263 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.773194    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-envoy\") pod \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") "
Oct 30 19:26:14.773263 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.773251    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-workload-certs\") pod \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") "
Oct 30 19:26:14.773471 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.773286    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-data\") pod \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") "
Oct 30 19:26:14.773471 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.773323    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-kube-api-access-gjjxp\" (UniqueName: \"kubernetes.io/projected/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-kube-api-access-gjjxp\") pod \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") "
Oct 30 19:26:14.773471 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.773361    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istiod-ca-cert\") pod \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") "
Oct 30 19:26:14.773471 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.773424    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-token\" (UniqueName: \"kubernetes.io/projected/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-token\") pod \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") "
Oct 30 19:26:14.773702 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.773507    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-podinfo\") pod \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") "
Oct 30 19:26:14.773702 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.773551    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-workload-socket\") pod \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") "
Oct 30 19:26:14.773702 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.773586    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-credential-socket\") pod \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\" (UID: \"6e08ca0d-5e22-4a83-b59e-e06c5ec74877\") "
Oct 30 19:26:14.773702 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.773687    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-workload-certs" pod "6e08ca0d-5e22-4a83-b59e-e06c5ec74877" (UID: "6e08ca0d-5e22-4a83-b59e-e06c5ec74877"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:14.774072 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.773953    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-credential-socket" pod "6e08ca0d-5e22-4a83-b59e-e06c5ec74877" (UID: "6e08ca0d-5e22-4a83-b59e-e06c5ec74877"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:14.774072 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.773970    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-workload-socket" pod "6e08ca0d-5e22-4a83-b59e-e06c5ec74877" (UID: "6e08ca0d-5e22-4a83-b59e-e06c5ec74877"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:14.774072 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.773979    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:14.774072 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.774065    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-data" pod "6e08ca0d-5e22-4a83-b59e-e06c5ec74877" (UID: "6e08ca0d-5e22-4a83-b59e-e06c5ec74877"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:14.774605 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.774283    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istiod-ca-cert" pod "6e08ca0d-5e22-4a83-b59e-e06c5ec74877" (UID: "6e08ca0d-5e22-4a83-b59e-e06c5ec74877"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:14.776285 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.776255    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-token" pod "6e08ca0d-5e22-4a83-b59e-e06c5ec74877" (UID: "6e08ca0d-5e22-4a83-b59e-e06c5ec74877"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:14.777242 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.776648    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-podinfo" pod "6e08ca0d-5e22-4a83-b59e-e06c5ec74877" (UID: "6e08ca0d-5e22-4a83-b59e-e06c5ec74877"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:26:14.777242 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.776680    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-kube-api-access-gjjxp" pod "6e08ca0d-5e22-4a83-b59e-e06c5ec74877" (UID: "6e08ca0d-5e22-4a83-b59e-e06c5ec74877"). InnerVolumeSpecName "kube-api-access-gjjxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:14.778175 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.778153    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-envoy" pod "6e08ca0d-5e22-4a83-b59e-e06c5ec74877" (UID: "6e08ca0d-5e22-4a83-b59e-e06c5ec74877"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:14.874532 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.874500    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:14.874532 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.874524    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:14.874532 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.874533    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:14.874708 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.874544    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:14.874708 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.874554    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjjxp\" (UniqueName: \"kubernetes.io/projected/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-kube-api-access-gjjxp\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:14.874708 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.874564    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:14.874708 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.874574    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:14.874708 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.874583    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e08ca0d-5e22-4a83-b59e-e06c5ec74877-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:14.934498 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.934472    4372 generic.go:356] "Generic (PLEG): container finished" podID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" containerID="364d35d7b9e06276c055d23db0127074c51b3077753da6e1510656852945f6bf" exitCode=0
Oct 30 19:26:14.934613 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.934526    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"
Oct 30 19:26:14.934613 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.934561    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g" event={"ID":"6e08ca0d-5e22-4a83-b59e-e06c5ec74877","Type":"ContainerDied","Data":"364d35d7b9e06276c055d23db0127074c51b3077753da6e1510656852945f6bf"}
Oct 30 19:26:14.934699 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.934621    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g" event={"ID":"6e08ca0d-5e22-4a83-b59e-e06c5ec74877","Type":"ContainerDied","Data":"4c790e9b3ffcba4f932d7e06f7ab312d192e1f51c3b07b0cfa94530a68c917be"}
Oct 30 19:26:14.934699 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.934654    4372 scope.go:122] "RemoveContainer" containerID="364d35d7b9e06276c055d23db0127074c51b3077753da6e1510656852945f6bf"
Oct 30 19:26:14.946663 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.946579    4372 scope.go:122] "RemoveContainer" containerID="6eef6f48881c32fa2b977112f8d09949d4d19d6742aa496957f2eeaa94f635f6"
Oct 30 19:26:14.955173 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.955152    4372 scope.go:122] "RemoveContainer" containerID="364d35d7b9e06276c055d23db0127074c51b3077753da6e1510656852945f6bf"
Oct 30 19:26:14.955440 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:26:14.955422    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"364d35d7b9e06276c055d23db0127074c51b3077753da6e1510656852945f6bf\": container with ID starting with 364d35d7b9e06276c055d23db0127074c51b3077753da6e1510656852945f6bf not found: ID does not exist" containerID="364d35d7b9e06276c055d23db0127074c51b3077753da6e1510656852945f6bf"
Oct 30 19:26:14.955495 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.955448    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"364d35d7b9e06276c055d23db0127074c51b3077753da6e1510656852945f6bf"} err="failed to get container status \"364d35d7b9e06276c055d23db0127074c51b3077753da6e1510656852945f6bf\": rpc error: code = NotFound desc = could not find container \"364d35d7b9e06276c055d23db0127074c51b3077753da6e1510656852945f6bf\": container with ID starting with 364d35d7b9e06276c055d23db0127074c51b3077753da6e1510656852945f6bf not found: ID does not exist"
Oct 30 19:26:14.955495 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.955463    4372 scope.go:122] "RemoveContainer" containerID="6eef6f48881c32fa2b977112f8d09949d4d19d6742aa496957f2eeaa94f635f6"
Oct 30 19:26:14.955671 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:26:14.955655    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eef6f48881c32fa2b977112f8d09949d4d19d6742aa496957f2eeaa94f635f6\": container with ID starting with 6eef6f48881c32fa2b977112f8d09949d4d19d6742aa496957f2eeaa94f635f6 not found: ID does not exist" containerID="6eef6f48881c32fa2b977112f8d09949d4d19d6742aa496957f2eeaa94f635f6"
Oct 30 19:26:14.955716 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.955674    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eef6f48881c32fa2b977112f8d09949d4d19d6742aa496957f2eeaa94f635f6"} err="failed to get container status \"6eef6f48881c32fa2b977112f8d09949d4d19d6742aa496957f2eeaa94f635f6\": rpc error: code = NotFound desc = could not find container \"6eef6f48881c32fa2b977112f8d09949d4d19d6742aa496957f2eeaa94f635f6\": container with ID starting with 6eef6f48881c32fa2b977112f8d09949d4d19d6742aa496957f2eeaa94f635f6 not found: ID does not exist"
Oct 30 19:26:14.958984 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.958959    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"]
Oct 30 19:26:14.963823 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:14.963802    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-58bb38-openshift-default-5f9f4c9485-sb79g"]
Oct 30 19:26:16.265554 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:16.265519    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" path="/var/lib/kubelet/pods/6e08ca0d-5e22-4a83-b59e-e06c5ec74877/volumes"
Oct 30 19:26:21.947979 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:21.947954    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"
Oct 30 19:26:21.976979 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:21.976910    4372 generic.go:356] "Generic (PLEG): container finished" podID="d49ae273-a4bf-4aae-bbac-6ed7c086f024" containerID="42b1561c2e5e2224c93a3f530c77cecc6d00bf7915d7694111a89c8117aec5a2" exitCode=0
Oct 30 19:26:21.976979 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:21.976964    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"
Oct 30 19:26:21.977197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:21.977004    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m" event={"ID":"d49ae273-a4bf-4aae-bbac-6ed7c086f024","Type":"ContainerDied","Data":"42b1561c2e5e2224c93a3f530c77cecc6d00bf7915d7694111a89c8117aec5a2"}
Oct 30 19:26:21.977197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:21.977057    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m" event={"ID":"d49ae273-a4bf-4aae-bbac-6ed7c086f024","Type":"ContainerDied","Data":"286c4d0e8d72d0947b58cf1a0f468f308a0703839e7382739c6af107cb8fa70b"}
Oct 30 19:26:21.977197 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:21.977084    4372 scope.go:122] "RemoveContainer" containerID="42b1561c2e5e2224c93a3f530c77cecc6d00bf7915d7694111a89c8117aec5a2"
Oct 30 19:26:21.988293 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:21.988276    4372 scope.go:122] "RemoveContainer" containerID="42b1561c2e5e2224c93a3f530c77cecc6d00bf7915d7694111a89c8117aec5a2"
Oct 30 19:26:21.988567 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:26:21.988550    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b1561c2e5e2224c93a3f530c77cecc6d00bf7915d7694111a89c8117aec5a2\": container with ID starting with 42b1561c2e5e2224c93a3f530c77cecc6d00bf7915d7694111a89c8117aec5a2 not found: ID does not exist" containerID="42b1561c2e5e2224c93a3f530c77cecc6d00bf7915d7694111a89c8117aec5a2"
Oct 30 19:26:21.988632 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:21.988578    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b1561c2e5e2224c93a3f530c77cecc6d00bf7915d7694111a89c8117aec5a2"} err="failed to get container status \"42b1561c2e5e2224c93a3f530c77cecc6d00bf7915d7694111a89c8117aec5a2\": rpc error: code = NotFound desc = could not find container \"42b1561c2e5e2224c93a3f530c77cecc6d00bf7915d7694111a89c8117aec5a2\": container with ID starting with 42b1561c2e5e2224c93a3f530c77cecc6d00bf7915d7694111a89c8117aec5a2 not found: ID does not exist"
Oct 30 19:26:22.038637 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:22.038614    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d49ae273-a4bf-4aae-bbac-6ed7c086f024-kube-api-access-p78ph\" (UniqueName: \"kubernetes.io/projected/d49ae273-a4bf-4aae-bbac-6ed7c086f024-kube-api-access-p78ph\") pod \"d49ae273-a4bf-4aae-bbac-6ed7c086f024\" (UID: \"d49ae273-a4bf-4aae-bbac-6ed7c086f024\") "
Oct 30 19:26:22.038749 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:22.038680    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d49ae273-a4bf-4aae-bbac-6ed7c086f024-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d49ae273-a4bf-4aae-bbac-6ed7c086f024-plugins-config-volume\") pod \"d49ae273-a4bf-4aae-bbac-6ed7c086f024\" (UID: \"d49ae273-a4bf-4aae-bbac-6ed7c086f024\") "
Oct 30 19:26:22.039027 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:22.039006    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49ae273-a4bf-4aae-bbac-6ed7c086f024-plugins-config-volume" pod "d49ae273-a4bf-4aae-bbac-6ed7c086f024" (UID: "d49ae273-a4bf-4aae-bbac-6ed7c086f024"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:22.040845 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:22.040824    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49ae273-a4bf-4aae-bbac-6ed7c086f024-kube-api-access-p78ph" pod "d49ae273-a4bf-4aae-bbac-6ed7c086f024" (UID: "d49ae273-a4bf-4aae-bbac-6ed7c086f024"). InnerVolumeSpecName "kube-api-access-p78ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:22.140087 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:22.140048    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d49ae273-a4bf-4aae-bbac-6ed7c086f024-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:22.140087 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:22.140081    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p78ph\" (UniqueName: \"kubernetes.io/projected/d49ae273-a4bf-4aae-bbac-6ed7c086f024-kube-api-access-p78ph\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:22.296204 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:22.296156    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"]
Oct 30 19:26:22.301523 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:22.301499    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58bb38-57487fd44c-rpb7m"]
Oct 30 19:26:24.264718 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.264682    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d49ae273-a4bf-4aae-bbac-6ed7c086f024" path="/var/lib/kubelet/pods/d49ae273-a4bf-4aae-bbac-6ed7c086f024/volumes"
Oct 30 19:26:24.517653 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.517577    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"]
Oct 30 19:26:24.517818 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.517769    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb" podUID="3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7" containerName="maas-api" containerID="cri-o://91a9b747fd6f3f405b76c5bd822673a87f182b4dc158b41a867da6b7937e8718" gracePeriod=30
Oct 30 19:26:24.775189 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.775107    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:26:24.864098 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.864067    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-kube-api-access-6j2pv\" (UniqueName: \"kubernetes.io/projected/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-kube-api-access-6j2pv\") pod \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") "
Oct 30 19:26:24.864308 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.864112    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-metrics-certs\") pod \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") "
Oct 30 19:26:24.864308 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.864238    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-openshift-service-ca\") pod \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") "
Oct 30 19:26:24.864308 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.864268    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-maas-api-tls\") pod \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\" (UID: \"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7\") "
Oct 30 19:26:24.864621 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.864591    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-openshift-service-ca" pod "3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7" (UID: "3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:24.866637 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.866609    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-metrics-certs" pod "3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7" (UID: "3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:24.866775 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.866744    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-kube-api-access-6j2pv" pod "3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7" (UID: "3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7"). InnerVolumeSpecName "kube-api-access-6j2pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:24.866775 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.866693    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-maas-api-tls" pod "3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7" (UID: "3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:24.965098 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.965071    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j2pv\" (UniqueName: \"kubernetes.io/projected/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-kube-api-access-6j2pv\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:24.965098 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.965097    4372 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-metrics-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:24.965276 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.965107    4372 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-openshift-service-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:24.965276 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:24.965117    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7-maas-api-tls\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:25.001668 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:25.001632    4372 generic.go:356] "Generic (PLEG): container finished" podID="3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7" containerID="91a9b747fd6f3f405b76c5bd822673a87f182b4dc158b41a867da6b7937e8718" exitCode=0
Oct 30 19:26:25.001837 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:25.001699    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"
Oct 30 19:26:25.001837 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:25.001711    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb" event={"ID":"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7","Type":"ContainerDied","Data":"91a9b747fd6f3f405b76c5bd822673a87f182b4dc158b41a867da6b7937e8718"}
Oct 30 19:26:25.001837 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:25.001801    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb" event={"ID":"3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7","Type":"ContainerDied","Data":"119e52266dff1fc273be3a3444b9b89c7180bac1e4e013f05c382d8b713e2af1"}
Oct 30 19:26:25.001837 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:25.001830    4372 scope.go:122] "RemoveContainer" containerID="91a9b747fd6f3f405b76c5bd822673a87f182b4dc158b41a867da6b7937e8718"
Oct 30 19:26:25.011758 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:25.011738    4372 scope.go:122] "RemoveContainer" containerID="91a9b747fd6f3f405b76c5bd822673a87f182b4dc158b41a867da6b7937e8718"
Oct 30 19:26:25.012030 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:26:25.012013    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a9b747fd6f3f405b76c5bd822673a87f182b4dc158b41a867da6b7937e8718\": container with ID starting with 91a9b747fd6f3f405b76c5bd822673a87f182b4dc158b41a867da6b7937e8718 not found: ID does not exist" containerID="91a9b747fd6f3f405b76c5bd822673a87f182b4dc158b41a867da6b7937e8718"
Oct 30 19:26:25.012077 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:25.012040    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a9b747fd6f3f405b76c5bd822673a87f182b4dc158b41a867da6b7937e8718"} err="failed to get container status \"91a9b747fd6f3f405b76c5bd822673a87f182b4dc158b41a867da6b7937e8718\": rpc error: code = NotFound desc = could not find container \"91a9b747fd6f3f405b76c5bd822673a87f182b4dc158b41a867da6b7937e8718\": container with ID starting with 91a9b747fd6f3f405b76c5bd822673a87f182b4dc158b41a867da6b7937e8718 not found: ID does not exist"
Oct 30 19:26:25.028233 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:25.028176    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"]
Oct 30 19:26:25.032216 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:25.032193    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a717c7-5c7bbc7647-wwvjb"]
Oct 30 19:26:26.263924 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:26.263858    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7" path="/var/lib/kubelet/pods/3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7/volumes"
Oct 30 19:26:27.679485 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:27.679451    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:26:27.695961 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:26:27.695918    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:26:43.682067 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:43.682030    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"]
Oct 30 19:26:43.702597 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:43.702556    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"]
Oct 30 19:26:47.096593 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:47.096560    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:26:48.682757 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:48.682714    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb" podUID="becab9bd-103e-4d46-8e6c-fe8c86602dc5" containerName="payload-processing" containerID="cri-o://998924f08ed36e7eb0d9bfc9f71b3d827982c74a704951ad556ad2ebb7bf01ad" gracePeriod=25
Oct 30 19:26:48.703524 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:48.703486    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8" podUID="0fa40c2a-bd44-4da8-95b0-3f68cfd29335" containerName="payload-pre-processing" containerID="cri-o://079a434103b1b346a28cfc5a31482f12c374bf51316e9cffd404f8e356201f64" gracePeriod=25
Oct 30 19:26:52.651497 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:52.651463    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"]
Oct 30 19:26:52.651967 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:52.651648    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597" podUID="6668392b-be79-447f-b009-907ff82e0420" containerName="istio-proxy" containerID="cri-o://60dbcca6af1b5fe7ffeb44760ddfc842f4b2b0ef86fe2cfdcfd7297efc241d7b" gracePeriod=30
Oct 30 19:26:55.037801 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:55.037754    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8" podUID="0fa40c2a-bd44-4da8-95b0-3f68cfd29335" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:26:55.038245 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:55.037781    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb" podUID="becab9bd-103e-4d46-8e6c-fe8c86602dc5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:26:57.695271 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:26:57.695227    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:26:57.916538 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.916476    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:26:57.970552 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.970522    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6668392b-be79-447f-b009-907ff82e0420-istio-token\" (UniqueName: \"kubernetes.io/projected/6668392b-be79-447f-b009-907ff82e0420-istio-token\") pod \"6668392b-be79-447f-b009-907ff82e0420\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") "
Oct 30 19:26:57.970714 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.970574    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-istio-envoy\") pod \"6668392b-be79-447f-b009-907ff82e0420\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") "
Oct 30 19:26:57.970714 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.970603    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-workload-certs\") pod \"6668392b-be79-447f-b009-907ff82e0420\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") "
Oct 30 19:26:57.970714 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.970630    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-workload-socket\") pod \"6668392b-be79-447f-b009-907ff82e0420\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") "
Oct 30 19:26:57.970714 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.970683    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6668392b-be79-447f-b009-907ff82e0420-kube-api-access-dnvsb\" (UniqueName: \"kubernetes.io/projected/6668392b-be79-447f-b009-907ff82e0420-kube-api-access-dnvsb\") pod \"6668392b-be79-447f-b009-907ff82e0420\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") "
Oct 30 19:26:57.970714 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.970713    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-istio-data\") pod \"6668392b-be79-447f-b009-907ff82e0420\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") "
Oct 30 19:26:57.970994 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.970742    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6668392b-be79-447f-b009-907ff82e0420-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6668392b-be79-447f-b009-907ff82e0420-istio-podinfo\") pod \"6668392b-be79-447f-b009-907ff82e0420\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") "
Oct 30 19:26:57.970994 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.970810    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6668392b-be79-447f-b009-907ff82e0420-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6668392b-be79-447f-b009-907ff82e0420-istiod-ca-cert\") pod \"6668392b-be79-447f-b009-907ff82e0420\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") "
Oct 30 19:26:57.970994 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.970852    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-credential-socket\") pod \"6668392b-be79-447f-b009-907ff82e0420\" (UID: \"6668392b-be79-447f-b009-907ff82e0420\") "
Oct 30 19:26:57.971190 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.970986    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-workload-socket" pod "6668392b-be79-447f-b009-907ff82e0420" (UID: "6668392b-be79-447f-b009-907ff82e0420"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:57.971190 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.970998    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-workload-certs" pod "6668392b-be79-447f-b009-907ff82e0420" (UID: "6668392b-be79-447f-b009-907ff82e0420"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:57.971313 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.971232    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-credential-socket" pod "6668392b-be79-447f-b009-907ff82e0420" (UID: "6668392b-be79-447f-b009-907ff82e0420"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:57.971313 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.971249    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-istio-data" pod "6668392b-be79-447f-b009-907ff82e0420" (UID: "6668392b-be79-447f-b009-907ff82e0420"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:57.971474 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.971319    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:57.971474 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.971339    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:57.971642 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.971552    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6668392b-be79-447f-b009-907ff82e0420-istiod-ca-cert" pod "6668392b-be79-447f-b009-907ff82e0420" (UID: "6668392b-be79-447f-b009-907ff82e0420"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:57.973018 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.972990    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6668392b-be79-447f-b009-907ff82e0420-istio-token" pod "6668392b-be79-447f-b009-907ff82e0420" (UID: "6668392b-be79-447f-b009-907ff82e0420"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:57.973291 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.973273    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-istio-envoy" pod "6668392b-be79-447f-b009-907ff82e0420" (UID: "6668392b-be79-447f-b009-907ff82e0420"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:57.973370 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.973355    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6668392b-be79-447f-b009-907ff82e0420-istio-podinfo" pod "6668392b-be79-447f-b009-907ff82e0420" (UID: "6668392b-be79-447f-b009-907ff82e0420"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:26:57.973507 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:57.973490    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6668392b-be79-447f-b009-907ff82e0420-kube-api-access-dnvsb" pod "6668392b-be79-447f-b009-907ff82e0420" (UID: "6668392b-be79-447f-b009-907ff82e0420"). InnerVolumeSpecName "kube-api-access-dnvsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:58.072284 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.072249    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6668392b-be79-447f-b009-907ff82e0420-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.072284 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.072286    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.072284 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.072297    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6668392b-be79-447f-b009-907ff82e0420-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.072489 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.072305    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.072489 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.072314    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnvsb\" (UniqueName: \"kubernetes.io/projected/6668392b-be79-447f-b009-907ff82e0420-kube-api-access-dnvsb\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.072489 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.072324    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6668392b-be79-447f-b009-907ff82e0420-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.072489 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.072333    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6668392b-be79-447f-b009-907ff82e0420-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.157369 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.157336    4372 generic.go:356] "Generic (PLEG): container finished" podID="6668392b-be79-447f-b009-907ff82e0420" containerID="60dbcca6af1b5fe7ffeb44760ddfc842f4b2b0ef86fe2cfdcfd7297efc241d7b" exitCode=0
Oct 30 19:26:58.157484 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.157386    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"
Oct 30 19:26:58.157484 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.157397    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597" event={"ID":"6668392b-be79-447f-b009-907ff82e0420","Type":"ContainerDied","Data":"60dbcca6af1b5fe7ffeb44760ddfc842f4b2b0ef86fe2cfdcfd7297efc241d7b"}
Oct 30 19:26:58.157484 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.157451    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597" event={"ID":"6668392b-be79-447f-b009-907ff82e0420","Type":"ContainerDied","Data":"b378cb7f03e88f95ef852e987687c877032d433fef9cc0737659f47bb10d8dcc"}
Oct 30 19:26:58.157484 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.157479    4372 scope.go:122] "RemoveContainer" containerID="60dbcca6af1b5fe7ffeb44760ddfc842f4b2b0ef86fe2cfdcfd7297efc241d7b"
Oct 30 19:26:58.167187 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.167170    4372 scope.go:122] "RemoveContainer" containerID="60dbcca6af1b5fe7ffeb44760ddfc842f4b2b0ef86fe2cfdcfd7297efc241d7b"
Oct 30 19:26:58.167484 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:26:58.167460    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60dbcca6af1b5fe7ffeb44760ddfc842f4b2b0ef86fe2cfdcfd7297efc241d7b\": container with ID starting with 60dbcca6af1b5fe7ffeb44760ddfc842f4b2b0ef86fe2cfdcfd7297efc241d7b not found: ID does not exist" containerID="60dbcca6af1b5fe7ffeb44760ddfc842f4b2b0ef86fe2cfdcfd7297efc241d7b"
Oct 30 19:26:58.167557 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.167493    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60dbcca6af1b5fe7ffeb44760ddfc842f4b2b0ef86fe2cfdcfd7297efc241d7b"} err="failed to get container status \"60dbcca6af1b5fe7ffeb44760ddfc842f4b2b0ef86fe2cfdcfd7297efc241d7b\": rpc error: code = NotFound desc = could not find container \"60dbcca6af1b5fe7ffeb44760ddfc842f4b2b0ef86fe2cfdcfd7297efc241d7b\": container with ID starting with 60dbcca6af1b5fe7ffeb44760ddfc842f4b2b0ef86fe2cfdcfd7297efc241d7b not found: ID does not exist"
Oct 30 19:26:58.191544 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.191519    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"]
Oct 30 19:26:58.198429 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.198407    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-x9597"]
Oct 30 19:26:58.263490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:26:58.263466    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6668392b-be79-447f-b009-907ff82e0420" path="/var/lib/kubelet/pods/6668392b-be79-447f-b009-907ff82e0420/volumes"
Oct 30 19:27:05.037203 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:05.037160    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8" podUID="0fa40c2a-bd44-4da8-95b0-3f68cfd29335" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:27:05.037610 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:05.037168    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb" podUID="becab9bd-103e-4d46-8e6c-fe8c86602dc5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:27:08.951350 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:08.951294    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"
Oct 30 19:27:08.981531 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:08.981505    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"
Oct 30 19:27:09.066189 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.066161    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/becab9bd-103e-4d46-8e6c-fe8c86602dc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/becab9bd-103e-4d46-8e6c-fe8c86602dc5-plugins-config-volume\") pod \"becab9bd-103e-4d46-8e6c-fe8c86602dc5\" (UID: \"becab9bd-103e-4d46-8e6c-fe8c86602dc5\") "
Oct 30 19:27:09.066312 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.066270    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa40c2a-bd44-4da8-95b0-3f68cfd29335-kube-api-access-t4sd5\" (UniqueName: \"kubernetes.io/projected/0fa40c2a-bd44-4da8-95b0-3f68cfd29335-kube-api-access-t4sd5\") pod \"0fa40c2a-bd44-4da8-95b0-3f68cfd29335\" (UID: \"0fa40c2a-bd44-4da8-95b0-3f68cfd29335\") "
Oct 30 19:27:09.066364 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.066338    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa40c2a-bd44-4da8-95b0-3f68cfd29335-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa40c2a-bd44-4da8-95b0-3f68cfd29335-plugins-config-volume\") pod \"0fa40c2a-bd44-4da8-95b0-3f68cfd29335\" (UID: \"0fa40c2a-bd44-4da8-95b0-3f68cfd29335\") "
Oct 30 19:27:09.066415 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.066375    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/becab9bd-103e-4d46-8e6c-fe8c86602dc5-kube-api-access-786kb\" (UniqueName: \"kubernetes.io/projected/becab9bd-103e-4d46-8e6c-fe8c86602dc5-kube-api-access-786kb\") pod \"becab9bd-103e-4d46-8e6c-fe8c86602dc5\" (UID: \"becab9bd-103e-4d46-8e6c-fe8c86602dc5\") "
Oct 30 19:27:09.066518 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.066492    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/becab9bd-103e-4d46-8e6c-fe8c86602dc5-plugins-config-volume" pod "becab9bd-103e-4d46-8e6c-fe8c86602dc5" (UID: "becab9bd-103e-4d46-8e6c-fe8c86602dc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:09.066727 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.066708    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/becab9bd-103e-4d46-8e6c-fe8c86602dc5-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:09.066789 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.066764    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa40c2a-bd44-4da8-95b0-3f68cfd29335-plugins-config-volume" pod "0fa40c2a-bd44-4da8-95b0-3f68cfd29335" (UID: "0fa40c2a-bd44-4da8-95b0-3f68cfd29335"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:09.068623 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.068604    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/becab9bd-103e-4d46-8e6c-fe8c86602dc5-kube-api-access-786kb" pod "becab9bd-103e-4d46-8e6c-fe8c86602dc5" (UID: "becab9bd-103e-4d46-8e6c-fe8c86602dc5"). InnerVolumeSpecName "kube-api-access-786kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:09.068683 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.068655    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa40c2a-bd44-4da8-95b0-3f68cfd29335-kube-api-access-t4sd5" pod "0fa40c2a-bd44-4da8-95b0-3f68cfd29335" (UID: "0fa40c2a-bd44-4da8-95b0-3f68cfd29335"). InnerVolumeSpecName "kube-api-access-t4sd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:09.167174 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.167150    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-786kb\" (UniqueName: \"kubernetes.io/projected/becab9bd-103e-4d46-8e6c-fe8c86602dc5-kube-api-access-786kb\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:09.167174 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.167174    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4sd5\" (UniqueName: \"kubernetes.io/projected/0fa40c2a-bd44-4da8-95b0-3f68cfd29335-kube-api-access-t4sd5\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:09.167324 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.167183    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa40c2a-bd44-4da8-95b0-3f68cfd29335-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:09.213699 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.213627    4372 generic.go:356] "Generic (PLEG): container finished" podID="0fa40c2a-bd44-4da8-95b0-3f68cfd29335" containerID="079a434103b1b346a28cfc5a31482f12c374bf51316e9cffd404f8e356201f64" exitCode=0
Oct 30 19:27:09.213699 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.213658    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8" event={"ID":"0fa40c2a-bd44-4da8-95b0-3f68cfd29335","Type":"ContainerDied","Data":"079a434103b1b346a28cfc5a31482f12c374bf51316e9cffd404f8e356201f64"}
Oct 30 19:27:09.213699 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.213677    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"
Oct 30 19:27:09.213933 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.213719    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8" event={"ID":"0fa40c2a-bd44-4da8-95b0-3f68cfd29335","Type":"ContainerDied","Data":"ecb68ebb0ec142a135c6cad764684fa8fbee8ab5ac82aecf19d36be325b747b7"}
Oct 30 19:27:09.213933 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.213747    4372 scope.go:122] "RemoveContainer" containerID="079a434103b1b346a28cfc5a31482f12c374bf51316e9cffd404f8e356201f64"
Oct 30 19:27:09.215395 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.215371    4372 generic.go:356] "Generic (PLEG): container finished" podID="becab9bd-103e-4d46-8e6c-fe8c86602dc5" containerID="998924f08ed36e7eb0d9bfc9f71b3d827982c74a704951ad556ad2ebb7bf01ad" exitCode=0
Oct 30 19:27:09.215528 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.215396    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb" event={"ID":"becab9bd-103e-4d46-8e6c-fe8c86602dc5","Type":"ContainerDied","Data":"998924f08ed36e7eb0d9bfc9f71b3d827982c74a704951ad556ad2ebb7bf01ad"}
Oct 30 19:27:09.215528 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.215414    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"
Oct 30 19:27:09.215528 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.215442    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb" event={"ID":"becab9bd-103e-4d46-8e6c-fe8c86602dc5","Type":"ContainerDied","Data":"7e003da04233b370c0a0c883791b187b83e4ab4b2b8d2915518036fdc0aea588"}
Oct 30 19:27:09.227646 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.227623    4372 scope.go:122] "RemoveContainer" containerID="079a434103b1b346a28cfc5a31482f12c374bf51316e9cffd404f8e356201f64"
Oct 30 19:27:09.227909 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:27:09.227886    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079a434103b1b346a28cfc5a31482f12c374bf51316e9cffd404f8e356201f64\": container with ID starting with 079a434103b1b346a28cfc5a31482f12c374bf51316e9cffd404f8e356201f64 not found: ID does not exist" containerID="079a434103b1b346a28cfc5a31482f12c374bf51316e9cffd404f8e356201f64"
Oct 30 19:27:09.227986 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.227917    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079a434103b1b346a28cfc5a31482f12c374bf51316e9cffd404f8e356201f64"} err="failed to get container status \"079a434103b1b346a28cfc5a31482f12c374bf51316e9cffd404f8e356201f64\": rpc error: code = NotFound desc = could not find container \"079a434103b1b346a28cfc5a31482f12c374bf51316e9cffd404f8e356201f64\": container with ID starting with 079a434103b1b346a28cfc5a31482f12c374bf51316e9cffd404f8e356201f64 not found: ID does not exist"
Oct 30 19:27:09.227986 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.227932    4372 scope.go:122] "RemoveContainer" containerID="998924f08ed36e7eb0d9bfc9f71b3d827982c74a704951ad556ad2ebb7bf01ad"
Oct 30 19:27:09.236244 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.236226    4372 scope.go:122] "RemoveContainer" containerID="998924f08ed36e7eb0d9bfc9f71b3d827982c74a704951ad556ad2ebb7bf01ad"
Oct 30 19:27:09.236500 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:27:09.236481    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998924f08ed36e7eb0d9bfc9f71b3d827982c74a704951ad556ad2ebb7bf01ad\": container with ID starting with 998924f08ed36e7eb0d9bfc9f71b3d827982c74a704951ad556ad2ebb7bf01ad not found: ID does not exist" containerID="998924f08ed36e7eb0d9bfc9f71b3d827982c74a704951ad556ad2ebb7bf01ad"
Oct 30 19:27:09.236551 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.236508    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998924f08ed36e7eb0d9bfc9f71b3d827982c74a704951ad556ad2ebb7bf01ad"} err="failed to get container status \"998924f08ed36e7eb0d9bfc9f71b3d827982c74a704951ad556ad2ebb7bf01ad\": rpc error: code = NotFound desc = could not find container \"998924f08ed36e7eb0d9bfc9f71b3d827982c74a704951ad556ad2ebb7bf01ad\": container with ID starting with 998924f08ed36e7eb0d9bfc9f71b3d827982c74a704951ad556ad2ebb7bf01ad not found: ID does not exist"
Oct 30 19:27:09.245482 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.245462    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"]
Oct 30 19:27:09.254472 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.254443    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f7c6e3-7dcf8fbd58-4bnzb"]
Oct 30 19:27:09.266735 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.266714    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"]
Oct 30 19:27:09.271862 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.271836    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f7c6e3-fdc9647ff-kqhl8"]
Oct 30 19:27:09.485370 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:09.485286    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:27:10.263718 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:10.263685    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa40c2a-bd44-4da8-95b0-3f68cfd29335" path="/var/lib/kubelet/pods/0fa40c2a-bd44-4da8-95b0-3f68cfd29335/volumes"
Oct 30 19:27:10.264116 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:10.264075    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="becab9bd-103e-4d46-8e6c-fe8c86602dc5" path="/var/lib/kubelet/pods/becab9bd-103e-4d46-8e6c-fe8c86602dc5/volumes"
Oct 30 19:27:27.696626 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:27:27.696579    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:27:28.754651 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:28.754615    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:27:40.671149 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:40.671081    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"]
Oct 30 19:27:46.124078 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.124040    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"]
Oct 30 19:27:46.930991 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.930956    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"]
Oct 30 19:27:46.931339 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931325    4372 cpu_manager.go:414] "containerMap: removing container" podUID="6668392b-be79-447f-b009-907ff82e0420" containerName="istio-proxy"
Oct 30 19:27:46.931399 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931341    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6668392b-be79-447f-b009-907ff82e0420" containerName="istio-proxy"
Oct 30 19:27:46.931399 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931354    4372 cpu_manager.go:414] "containerMap: removing container" podUID="0f86acaf-a6dd-4962-82db-f2103f9f263f" containerName="maas-api"
Oct 30 19:27:46.931399 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931358    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f86acaf-a6dd-4962-82db-f2103f9f263f" containerName="maas-api"
Oct 30 19:27:46.931399 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931373    4372 cpu_manager.go:414] "containerMap: removing container" podUID="8144c849-ae43-4536-b9d1-4dbdde207847" containerName="maas-api"
Oct 30 19:27:46.931399 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931378    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8144c849-ae43-4536-b9d1-4dbdde207847" containerName="maas-api"
Oct 30 19:27:46.931399 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931385    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" containerName="istio-proxy"
Oct 30 19:27:46.931399 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931390    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" containerName="istio-proxy"
Oct 30 19:27:46.931399 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931395    4372 cpu_manager.go:414] "containerMap: removing container" podUID="34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665" containerName="payload-processing"
Oct 30 19:27:46.931399 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931400    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665" containerName="payload-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931411    4372 cpu_manager.go:414] "containerMap: removing container" podUID="becab9bd-103e-4d46-8e6c-fe8c86602dc5" containerName="payload-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931416    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="becab9bd-103e-4d46-8e6c-fe8c86602dc5" containerName="payload-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931425    4372 cpu_manager.go:414] "containerMap: removing container" podUID="33dfa4ab-12fb-4444-b419-9c24113ea33e" containerName="payload-pre-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931430    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33dfa4ab-12fb-4444-b419-9c24113ea33e" containerName="payload-pre-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931438    4372 cpu_manager.go:414] "containerMap: removing container" podUID="ba2afdc1-190a-4aea-9cc9-2692bcaf56c7" containerName="main"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931442    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba2afdc1-190a-4aea-9cc9-2692bcaf56c7" containerName="main"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931449    4372 cpu_manager.go:414] "containerMap: removing container" podUID="29099d2d-89eb-49da-b9ea-7e9c4914d571" containerName="main"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931453    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29099d2d-89eb-49da-b9ea-7e9c4914d571" containerName="main"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931460    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d89c0e0e-7b58-4c7f-a959-75e56c711f6f" containerName="payload-pre-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931466    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89c0e0e-7b58-4c7f-a959-75e56c711f6f" containerName="payload-pre-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931473    4372 cpu_manager.go:414] "containerMap: removing container" podUID="450a9e2f-36a4-419e-a586-981cda482d97" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931477    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450a9e2f-36a4-419e-a586-981cda482d97" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931486    4372 cpu_manager.go:414] "containerMap: removing container" podUID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931491    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931499    4372 cpu_manager.go:414] "containerMap: removing container" podUID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931503    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931511    4372 cpu_manager.go:414] "containerMap: removing container" podUID="72834257-61fb-4596-8b82-5917990f7469" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931515    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72834257-61fb-4596-8b82-5917990f7469" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931521    4372 cpu_manager.go:414] "containerMap: removing container" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931525    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931532    4372 cpu_manager.go:414] "containerMap: removing container" podUID="450a9e2f-36a4-419e-a586-981cda482d97" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931537    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450a9e2f-36a4-419e-a586-981cda482d97" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931543    4372 cpu_manager.go:414] "containerMap: removing container" podUID="d49ae273-a4bf-4aae-bbac-6ed7c086f024" containerName="payload-pre-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931547    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d49ae273-a4bf-4aae-bbac-6ed7c086f024" containerName="payload-pre-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931559    4372 cpu_manager.go:414] "containerMap: removing container" podUID="3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7" containerName="maas-api"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931563    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7" containerName="maas-api"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931570    4372 cpu_manager.go:414] "containerMap: removing container" podUID="211df401-6e2e-4db2-a0c9-4e0d24532573" containerName="payload-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931574    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211df401-6e2e-4db2-a0c9-4e0d24532573" containerName="payload-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931581    4372 cpu_manager.go:414] "containerMap: removing container" podUID="72834257-61fb-4596-8b82-5917990f7469" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931585    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72834257-61fb-4596-8b82-5917990f7469" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931590    4372 cpu_manager.go:414] "containerMap: removing container" podUID="0fa40c2a-bd44-4da8-95b0-3f68cfd29335" containerName="payload-pre-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931595    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa40c2a-bd44-4da8-95b0-3f68cfd29335" containerName="payload-pre-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931647    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931656    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e08ca0d-5e22-4a83-b59e-e06c5ec74877" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931663    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="72834257-61fb-4596-8b82-5917990f7469" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931670    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="665d8ad1-52bf-4837-9a43-c21c4b2ea5ab" containerName="main"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931675    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="450a9e2f-36a4-419e-a586-981cda482d97" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931679    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d49ae273-a4bf-4aae-bbac-6ed7c086f024" containerName="payload-pre-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931686    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae7c57e-6d9f-4fa1-a9cc-0256ba10fab7" containerName="maas-api"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931691    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="211df401-6e2e-4db2-a0c9-4e0d24532573" containerName="payload-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931695    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa40c2a-bd44-4da8-95b0-3f68cfd29335" containerName="payload-pre-processing"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931701    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="6668392b-be79-447f-b009-907ff82e0420" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931706    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f86acaf-a6dd-4962-82db-f2103f9f263f" containerName="maas-api"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931711    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="8144c849-ae43-4536-b9d1-4dbdde207847" containerName="maas-api"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931716    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10d45d9-3b28-4c0e-aeb6-62a63f39ab33" containerName="istio-proxy"
Oct 30 19:27:46.931694 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931722    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ce7f8a-dc9d-4b3e-b75d-a5d0f665b665" containerName="payload-processing"
Oct 30 19:27:46.933210 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931727    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="becab9bd-103e-4d46-8e6c-fe8c86602dc5" containerName="payload-processing"
Oct 30 19:27:46.933210 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931733    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="33dfa4ab-12fb-4444-b419-9c24113ea33e" containerName="payload-pre-processing"
Oct 30 19:27:46.933210 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931738    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba2afdc1-190a-4aea-9cc9-2692bcaf56c7" containerName="main"
Oct 30 19:27:46.933210 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931743    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="29099d2d-89eb-49da-b9ea-7e9c4914d571" containerName="main"
Oct 30 19:27:46.933210 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931748    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89c0e0e-7b58-4c7f-a959-75e56c711f6f" containerName="payload-pre-processing"
Oct 30 19:27:46.933210 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.931753    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="450a9e2f-36a4-419e-a586-981cda482d97" containerName="istio-proxy"
Oct 30 19:27:46.934742 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.934724    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"
Oct 30 19:27:46.950534 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.950505    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"]
Oct 30 19:27:46.986729 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.986652    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c0b8fad-f840-43ec-b061-220e42012129-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4af4b4e7-9hhml\" (UID: \"6c0b8fad-f840-43ec-b061-220e42012129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"
Oct 30 19:27:46.986729 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.986695    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c0b8fad-f840-43ec-b061-220e42012129-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4af4b4e7-9hhml\" (UID: \"6c0b8fad-f840-43ec-b061-220e42012129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"
Oct 30 19:27:46.986952 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:46.986891    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psxrs\" (UniqueName: \"kubernetes.io/projected/6c0b8fad-f840-43ec-b061-220e42012129-kube-api-access-psxrs\") pod \"maas-api-revoke-sub-4af4b4e7-9hhml\" (UID: \"6c0b8fad-f840-43ec-b061-220e42012129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"
Oct 30 19:27:47.041690 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:47.041667    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:27:47.042024 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:47.042007    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\""
Oct 30 19:27:47.087705 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:47.087665    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psxrs\" (UniqueName: \"kubernetes.io/projected/6c0b8fad-f840-43ec-b061-220e42012129-kube-api-access-psxrs\") pod \"maas-api-revoke-sub-4af4b4e7-9hhml\" (UID: \"6c0b8fad-f840-43ec-b061-220e42012129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"
Oct 30 19:27:47.087839 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:47.087735    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c0b8fad-f840-43ec-b061-220e42012129-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4af4b4e7-9hhml\" (UID: \"6c0b8fad-f840-43ec-b061-220e42012129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"
Oct 30 19:27:47.087839 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:47.087767    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c0b8fad-f840-43ec-b061-220e42012129-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4af4b4e7-9hhml\" (UID: \"6c0b8fad-f840-43ec-b061-220e42012129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"
Oct 30 19:27:47.088466 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:47.088446    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c0b8fad-f840-43ec-b061-220e42012129-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4af4b4e7-9hhml\" (UID: \"6c0b8fad-f840-43ec-b061-220e42012129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"
Oct 30 19:27:47.090501 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:47.090476    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c0b8fad-f840-43ec-b061-220e42012129-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4af4b4e7-9hhml\" (UID: \"6c0b8fad-f840-43ec-b061-220e42012129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"
Oct 30 19:27:47.101114 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:47.101087    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psxrs\" (UniqueName: \"kubernetes.io/projected/6c0b8fad-f840-43ec-b061-220e42012129-kube-api-access-psxrs\") pod \"maas-api-revoke-sub-4af4b4e7-9hhml\" (UID: \"6c0b8fad-f840-43ec-b061-220e42012129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"
Oct 30 19:27:47.247710 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:47.247651    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"
Oct 30 19:27:47.523499 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:47.523472    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"]
Oct 30 19:27:47.525983 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:27:47.525949    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0b8fad_f840_43ec_b061_220e42012129.slice/crio-9802a11d16f863b7af271db0a06efc5ddcd782b87341a93bf0ffb7e703d31e21 WatchSource:0}: Error finding container 9802a11d16f863b7af271db0a06efc5ddcd782b87341a93bf0ffb7e703d31e21: Status 404 returned error can't find the container with id 9802a11d16f863b7af271db0a06efc5ddcd782b87341a93bf0ffb7e703d31e21
Oct 30 19:27:48.389765 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:48.389737    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml" event={"ID":"6c0b8fad-f840-43ec-b061-220e42012129","Type":"ContainerStarted","Data":"b1c409988e4f95e4171721990d77fc1452bad600f4db7c48850975f65fe1e218"}
Oct 30 19:27:48.390087 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:48.389790    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml" event={"ID":"6c0b8fad-f840-43ec-b061-220e42012129","Type":"ContainerStarted","Data":"9802a11d16f863b7af271db0a06efc5ddcd782b87341a93bf0ffb7e703d31e21"}
Oct 30 19:27:49.395029 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:49.394993    4372 generic.go:356] "Generic (PLEG): container finished" podID="6c0b8fad-f840-43ec-b061-220e42012129" containerID="b1c409988e4f95e4171721990d77fc1452bad600f4db7c48850975f65fe1e218" exitCode=0
Oct 30 19:27:49.395470 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:49.395067    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml" event={"ID":"6c0b8fad-f840-43ec-b061-220e42012129","Type":"ContainerDied","Data":"b1c409988e4f95e4171721990d77fc1452bad600f4db7c48850975f65fe1e218"}
Oct 30 19:27:49.525644 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:49.525618    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"
Oct 30 19:27:49.609524 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:49.609497    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c0b8fad-f840-43ec-b061-220e42012129-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c0b8fad-f840-43ec-b061-220e42012129-maas-api-openshift-ca\") pod \"6c0b8fad-f840-43ec-b061-220e42012129\" (UID: \"6c0b8fad-f840-43ec-b061-220e42012129\") "
Oct 30 19:27:49.609637 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:49.609548    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c0b8fad-f840-43ec-b061-220e42012129-kube-api-access-psxrs\" (UniqueName: \"kubernetes.io/projected/6c0b8fad-f840-43ec-b061-220e42012129-kube-api-access-psxrs\") pod \"6c0b8fad-f840-43ec-b061-220e42012129\" (UID: \"6c0b8fad-f840-43ec-b061-220e42012129\") "
Oct 30 19:27:49.609691 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:49.609670    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c0b8fad-f840-43ec-b061-220e42012129-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c0b8fad-f840-43ec-b061-220e42012129-maas-api-xks-ca\") pod \"6c0b8fad-f840-43ec-b061-220e42012129\" (UID: \"6c0b8fad-f840-43ec-b061-220e42012129\") "
Oct 30 19:27:49.609925 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:49.609903    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c0b8fad-f840-43ec-b061-220e42012129-maas-api-openshift-ca" pod "6c0b8fad-f840-43ec-b061-220e42012129" (UID: "6c0b8fad-f840-43ec-b061-220e42012129"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:49.611895 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:49.611871    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c0b8fad-f840-43ec-b061-220e42012129-maas-api-xks-ca" pod "6c0b8fad-f840-43ec-b061-220e42012129" (UID: "6c0b8fad-f840-43ec-b061-220e42012129"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:49.611981 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:49.611898    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0b8fad-f840-43ec-b061-220e42012129-kube-api-access-psxrs" pod "6c0b8fad-f840-43ec-b061-220e42012129" (UID: "6c0b8fad-f840-43ec-b061-220e42012129"). InnerVolumeSpecName "kube-api-access-psxrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:49.710480 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:49.710426    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c0b8fad-f840-43ec-b061-220e42012129-maas-api-openshift-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:49.710480 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:49.710449    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psxrs\" (UniqueName: \"kubernetes.io/projected/6c0b8fad-f840-43ec-b061-220e42012129-kube-api-access-psxrs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:49.710480 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:49.710459    4372 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c0b8fad-f840-43ec-b061-220e42012129-maas-api-xks-ca\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:50.400415 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:50.400388    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml"
Oct 30 19:27:50.400800 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:50.400387    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4af4b4e7-9hhml" event={"ID":"6c0b8fad-f840-43ec-b061-220e42012129","Type":"ContainerDied","Data":"9802a11d16f863b7af271db0a06efc5ddcd782b87341a93bf0ffb7e703d31e21"}
Oct 30 19:27:50.400800 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:50.400510    4372 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9802a11d16f863b7af271db0a06efc5ddcd782b87341a93bf0ffb7e703d31e21"
Oct 30 19:27:51.124409 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:51.124340    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t" podUID="23924268-e8d8-4506-8581-5c5ae5fc9ddf" containerName="payload-processing" containerID="cri-o://4ca7ffd27e66e78dcd9c9b64a3e4871bd7d940c8e3cc1a55324ae431bc52eaa9" gracePeriod=25
Oct 30 19:27:53.718306 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:53.718266    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t" podUID="23924268-e8d8-4506-8581-5c5ae5fc9ddf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:27:55.691905 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:55.691859    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main" containerID="cri-o://26be516e7f8b230308d5de26b5d4f556923c00c5d104e5c8c5c6257937103d0a" gracePeriod=45
Oct 30 19:27:55.945288 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:55.945234    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 19:27:56.058576 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.058543    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-home\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-home\") pod \"22579f9b-ea36-4a4b-867b-87019a0b62c6\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") "
Oct 30 19:27:56.058767 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.058620    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22579f9b-ea36-4a4b-867b-87019a0b62c6-kube-api-access-4c272\" (UniqueName: \"kubernetes.io/projected/22579f9b-ea36-4a4b-867b-87019a0b62c6-kube-api-access-4c272\") pod \"22579f9b-ea36-4a4b-867b-87019a0b62c6\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") "
Oct 30 19:27:56.058767 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.058704    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-tmp-dir\") pod \"22579f9b-ea36-4a4b-867b-87019a0b62c6\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") "
Oct 30 19:27:56.058767 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.058731    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-model-cache\") pod \"22579f9b-ea36-4a4b-867b-87019a0b62c6\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") "
Oct 30 19:27:56.058767 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.058755    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-dshm\") pod \"22579f9b-ea36-4a4b-867b-87019a0b62c6\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") "
Oct 30 19:27:56.059007 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.058793    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22579f9b-ea36-4a4b-867b-87019a0b62c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/22579f9b-ea36-4a4b-867b-87019a0b62c6-tls-certs\") pod \"22579f9b-ea36-4a4b-867b-87019a0b62c6\" (UID: \"22579f9b-ea36-4a4b-867b-87019a0b62c6\") "
Oct 30 19:27:56.059007 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.058847    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-home" pod "22579f9b-ea36-4a4b-867b-87019a0b62c6" (UID: "22579f9b-ea36-4a4b-867b-87019a0b62c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:56.059116 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.059022    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-model-cache" pod "22579f9b-ea36-4a4b-867b-87019a0b62c6" (UID: "22579f9b-ea36-4a4b-867b-87019a0b62c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:56.059116 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.059036    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-tmp-dir" pod "22579f9b-ea36-4a4b-867b-87019a0b62c6" (UID: "22579f9b-ea36-4a4b-867b-87019a0b62c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:56.059231 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.059184    4372 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-home\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.059231 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.059201    4372 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-tmp-dir\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.059231 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.059210    4372 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-model-cache\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.061214 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.061178    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22579f9b-ea36-4a4b-867b-87019a0b62c6-tls-certs" pod "22579f9b-ea36-4a4b-867b-87019a0b62c6" (UID: "22579f9b-ea36-4a4b-867b-87019a0b62c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:56.061214 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.061196    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-dshm" pod "22579f9b-ea36-4a4b-867b-87019a0b62c6" (UID: "22579f9b-ea36-4a4b-867b-87019a0b62c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:56.061528 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.061505    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22579f9b-ea36-4a4b-867b-87019a0b62c6-kube-api-access-4c272" pod "22579f9b-ea36-4a4b-867b-87019a0b62c6" (UID: "22579f9b-ea36-4a4b-867b-87019a0b62c6"). InnerVolumeSpecName "kube-api-access-4c272". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:56.159869 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.159836    4372 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22579f9b-ea36-4a4b-867b-87019a0b62c6-dshm\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.159869 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.159864    4372 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22579f9b-ea36-4a4b-867b-87019a0b62c6-tls-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.159869 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.159874    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c272\" (UniqueName: \"kubernetes.io/projected/22579f9b-ea36-4a4b-867b-87019a0b62c6-kube-api-access-4c272\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.429388 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.429357    4372 generic.go:356] "Generic (PLEG): container finished" podID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerID="26be516e7f8b230308d5de26b5d4f556923c00c5d104e5c8c5c6257937103d0a" exitCode=0
Oct 30 19:27:56.429550 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.429408    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"
Oct 30 19:27:56.429550 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.429445    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" event={"ID":"22579f9b-ea36-4a4b-867b-87019a0b62c6","Type":"ContainerDied","Data":"26be516e7f8b230308d5de26b5d4f556923c00c5d104e5c8c5c6257937103d0a"}
Oct 30 19:27:56.429550 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.429498    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv" event={"ID":"22579f9b-ea36-4a4b-867b-87019a0b62c6","Type":"ContainerDied","Data":"e8251de367602a33c87b7d05de8d07cf07a795fba9b116f03b0dda89f0ca0d04"}
Oct 30 19:27:56.429550 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.429521    4372 scope.go:122] "RemoveContainer" containerID="26be516e7f8b230308d5de26b5d4f556923c00c5d104e5c8c5c6257937103d0a"
Oct 30 19:27:56.438564 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.438542    4372 scope.go:122] "RemoveContainer" containerID="26be516e7f8b230308d5de26b5d4f556923c00c5d104e5c8c5c6257937103d0a"
Oct 30 19:27:56.438880 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:27:56.438858    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26be516e7f8b230308d5de26b5d4f556923c00c5d104e5c8c5c6257937103d0a\": container with ID starting with 26be516e7f8b230308d5de26b5d4f556923c00c5d104e5c8c5c6257937103d0a not found: ID does not exist" containerID="26be516e7f8b230308d5de26b5d4f556923c00c5d104e5c8c5c6257937103d0a"
Oct 30 19:27:56.438959 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.438906    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26be516e7f8b230308d5de26b5d4f556923c00c5d104e5c8c5c6257937103d0a"} err="failed to get container status \"26be516e7f8b230308d5de26b5d4f556923c00c5d104e5c8c5c6257937103d0a\": rpc error: code = NotFound desc = could not find container \"26be516e7f8b230308d5de26b5d4f556923c00c5d104e5c8c5c6257937103d0a\": container with ID starting with 26be516e7f8b230308d5de26b5d4f556923c00c5d104e5c8c5c6257937103d0a not found: ID does not exist"
Oct 30 19:27:56.450054 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.450030    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"]
Oct 30 19:27:56.456613 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.456591    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/premium-w2-8a92cc-kserve-6797c69b66-jwbgv"]
Oct 30 19:27:56.855458 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.855423    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"]
Oct 30 19:27:56.855876 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:56.855694    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q" podUID="dd868454-02ed-4227-ae09-be713e3473d1" containerName="istio-proxy" containerID="cri-o://8f6b27eb0dfa4b8eae0c6d9ae94467a7bff03fd3249ab78071aed8d3ab6efc86" gracePeriod=30
Oct 30 19:27:57.698448 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:27:57.698407    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:27:58.262980 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:58.262946    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" path="/var/lib/kubelet/pods/22579f9b-ea36-4a4b-867b-87019a0b62c6/volumes"
Oct 30 19:27:59.069014 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:27:59.068963    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:28:02.110537 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.110511    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:28:02.208366 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.208285    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-workload-certs\") pod \"dd868454-02ed-4227-ae09-be713e3473d1\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") "
Oct 30 19:28:02.208366 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.208329    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-istio-data\") pod \"dd868454-02ed-4227-ae09-be713e3473d1\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") "
Oct 30 19:28:02.208366 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.208360    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd868454-02ed-4227-ae09-be713e3473d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd868454-02ed-4227-ae09-be713e3473d1-istiod-ca-cert\") pod \"dd868454-02ed-4227-ae09-be713e3473d1\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") "
Oct 30 19:28:02.208648 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.208394    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-credential-socket\") pod \"dd868454-02ed-4227-ae09-be713e3473d1\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") "
Oct 30 19:28:02.208648 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.208446    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd868454-02ed-4227-ae09-be713e3473d1-istio-token\" (UniqueName: \"kubernetes.io/projected/dd868454-02ed-4227-ae09-be713e3473d1-istio-token\") pod \"dd868454-02ed-4227-ae09-be713e3473d1\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") "
Oct 30 19:28:02.208648 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.208476    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-istio-envoy\") pod \"dd868454-02ed-4227-ae09-be713e3473d1\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") "
Oct 30 19:28:02.208648 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.208508    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd868454-02ed-4227-ae09-be713e3473d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd868454-02ed-4227-ae09-be713e3473d1-istio-podinfo\") pod \"dd868454-02ed-4227-ae09-be713e3473d1\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") "
Oct 30 19:28:02.208648 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.208555    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-workload-socket\") pod \"dd868454-02ed-4227-ae09-be713e3473d1\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") "
Oct 30 19:28:02.208648 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.208642    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd868454-02ed-4227-ae09-be713e3473d1-kube-api-access-qmmfj\" (UniqueName: \"kubernetes.io/projected/dd868454-02ed-4227-ae09-be713e3473d1-kube-api-access-qmmfj\") pod \"dd868454-02ed-4227-ae09-be713e3473d1\" (UID: \"dd868454-02ed-4227-ae09-be713e3473d1\") "
Oct 30 19:28:02.208982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.208666    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-istio-data" pod "dd868454-02ed-4227-ae09-be713e3473d1" (UID: "dd868454-02ed-4227-ae09-be713e3473d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:02.208982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.208688    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-workload-certs" pod "dd868454-02ed-4227-ae09-be713e3473d1" (UID: "dd868454-02ed-4227-ae09-be713e3473d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:02.208982 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.208852    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-credential-socket" pod "dd868454-02ed-4227-ae09-be713e3473d1" (UID: "dd868454-02ed-4227-ae09-be713e3473d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:02.209172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.208985    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-workload-socket" pod "dd868454-02ed-4227-ae09-be713e3473d1" (UID: "dd868454-02ed-4227-ae09-be713e3473d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:02.209172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.209019    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd868454-02ed-4227-ae09-be713e3473d1-istiod-ca-cert" pod "dd868454-02ed-4227-ae09-be713e3473d1" (UID: "dd868454-02ed-4227-ae09-be713e3473d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:02.209172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.209031    4372 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-workload-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:02.209172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.209054    4372 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-istio-data\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:02.209172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.209071    4372 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-credential-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:02.211041 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.211011    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd868454-02ed-4227-ae09-be713e3473d1-istio-podinfo" pod "dd868454-02ed-4227-ae09-be713e3473d1" (UID: "dd868454-02ed-4227-ae09-be713e3473d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:28:02.211187 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.211042    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd868454-02ed-4227-ae09-be713e3473d1-kube-api-access-qmmfj" pod "dd868454-02ed-4227-ae09-be713e3473d1" (UID: "dd868454-02ed-4227-ae09-be713e3473d1"). InnerVolumeSpecName "kube-api-access-qmmfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:02.211187 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.211115    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-istio-envoy" pod "dd868454-02ed-4227-ae09-be713e3473d1" (UID: "dd868454-02ed-4227-ae09-be713e3473d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:02.211187 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.211140    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd868454-02ed-4227-ae09-be713e3473d1-istio-token" pod "dd868454-02ed-4227-ae09-be713e3473d1" (UID: "dd868454-02ed-4227-ae09-be713e3473d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:02.310348 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.310323    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmmfj\" (UniqueName: \"kubernetes.io/projected/dd868454-02ed-4227-ae09-be713e3473d1-kube-api-access-qmmfj\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:02.310348 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.310349    4372 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd868454-02ed-4227-ae09-be713e3473d1-istiod-ca-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:02.310490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.310359    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd868454-02ed-4227-ae09-be713e3473d1-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:02.310490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.310368    4372 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-istio-envoy\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:02.310490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.310376    4372 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd868454-02ed-4227-ae09-be713e3473d1-istio-podinfo\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:02.310490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.310385    4372 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd868454-02ed-4227-ae09-be713e3473d1-workload-socket\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:02.460450 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.460362    4372 generic.go:356] "Generic (PLEG): container finished" podID="dd868454-02ed-4227-ae09-be713e3473d1" containerID="8f6b27eb0dfa4b8eae0c6d9ae94467a7bff03fd3249ab78071aed8d3ab6efc86" exitCode=0
Oct 30 19:28:02.460450 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.460413    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"
Oct 30 19:28:02.460705 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.460447    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q" event={"ID":"dd868454-02ed-4227-ae09-be713e3473d1","Type":"ContainerDied","Data":"8f6b27eb0dfa4b8eae0c6d9ae94467a7bff03fd3249ab78071aed8d3ab6efc86"}
Oct 30 19:28:02.460705 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.460512    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q" event={"ID":"dd868454-02ed-4227-ae09-be713e3473d1","Type":"ContainerDied","Data":"c914b9b1e221febbfc89f0db0cb9ccd05d3c5602cba51a67d9b226b9970565bc"}
Oct 30 19:28:02.460705 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.460541    4372 scope.go:122] "RemoveContainer" containerID="8f6b27eb0dfa4b8eae0c6d9ae94467a7bff03fd3249ab78071aed8d3ab6efc86"
Oct 30 19:28:02.469636 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.469619    4372 scope.go:122] "RemoveContainer" containerID="8f6b27eb0dfa4b8eae0c6d9ae94467a7bff03fd3249ab78071aed8d3ab6efc86"
Oct 30 19:28:02.469919 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:28:02.469900    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f6b27eb0dfa4b8eae0c6d9ae94467a7bff03fd3249ab78071aed8d3ab6efc86\": container with ID starting with 8f6b27eb0dfa4b8eae0c6d9ae94467a7bff03fd3249ab78071aed8d3ab6efc86 not found: ID does not exist" containerID="8f6b27eb0dfa4b8eae0c6d9ae94467a7bff03fd3249ab78071aed8d3ab6efc86"
Oct 30 19:28:02.469985 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.469927    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f6b27eb0dfa4b8eae0c6d9ae94467a7bff03fd3249ab78071aed8d3ab6efc86"} err="failed to get container status \"8f6b27eb0dfa4b8eae0c6d9ae94467a7bff03fd3249ab78071aed8d3ab6efc86\": rpc error: code = NotFound desc = could not find container \"8f6b27eb0dfa4b8eae0c6d9ae94467a7bff03fd3249ab78071aed8d3ab6efc86\": container with ID starting with 8f6b27eb0dfa4b8eae0c6d9ae94467a7bff03fd3249ab78071aed8d3ab6efc86 not found: ID does not exist"
Oct 30 19:28:02.479656 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.479630    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"]
Oct 30 19:28:02.490775 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:02.490752    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-bjk2q"]
Oct 30 19:28:03.718426 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:03.718380    4372 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t" podUID="23924268-e8d8-4506-8581-5c5ae5fc9ddf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:28:04.266159 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:04.266102    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd868454-02ed-4227-ae09-be713e3473d1" path="/var/lib/kubelet/pods/dd868454-02ed-4227-ae09-be713e3473d1/volumes"
Oct 30 19:28:07.342266 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:07.342219    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qz7ks"]
Oct 30 19:28:11.373139 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.373099    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"
Oct 30 19:28:11.492957 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.492888    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23924268-e8d8-4506-8581-5c5ae5fc9ddf-kube-api-access-2xp5z\" (UniqueName: \"kubernetes.io/projected/23924268-e8d8-4506-8581-5c5ae5fc9ddf-kube-api-access-2xp5z\") pod \"23924268-e8d8-4506-8581-5c5ae5fc9ddf\" (UID: \"23924268-e8d8-4506-8581-5c5ae5fc9ddf\") "
Oct 30 19:28:11.492957 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.492933    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23924268-e8d8-4506-8581-5c5ae5fc9ddf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23924268-e8d8-4506-8581-5c5ae5fc9ddf-plugins-config-volume\") pod \"23924268-e8d8-4506-8581-5c5ae5fc9ddf\" (UID: \"23924268-e8d8-4506-8581-5c5ae5fc9ddf\") "
Oct 30 19:28:11.493340 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.493318    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23924268-e8d8-4506-8581-5c5ae5fc9ddf-plugins-config-volume" pod "23924268-e8d8-4506-8581-5c5ae5fc9ddf" (UID: "23924268-e8d8-4506-8581-5c5ae5fc9ddf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:11.495186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.495163    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23924268-e8d8-4506-8581-5c5ae5fc9ddf-kube-api-access-2xp5z" pod "23924268-e8d8-4506-8581-5c5ae5fc9ddf" (UID: "23924268-e8d8-4506-8581-5c5ae5fc9ddf"). InnerVolumeSpecName "kube-api-access-2xp5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:11.504189 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.504166    4372 generic.go:356] "Generic (PLEG): container finished" podID="23924268-e8d8-4506-8581-5c5ae5fc9ddf" containerID="4ca7ffd27e66e78dcd9c9b64a3e4871bd7d940c8e3cc1a55324ae431bc52eaa9" exitCode=0
Oct 30 19:28:11.504323 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.504220    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"
Oct 30 19:28:11.504323 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.504245    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t" event={"ID":"23924268-e8d8-4506-8581-5c5ae5fc9ddf","Type":"ContainerDied","Data":"4ca7ffd27e66e78dcd9c9b64a3e4871bd7d940c8e3cc1a55324ae431bc52eaa9"}
Oct 30 19:28:11.504323 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.504304    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t" event={"ID":"23924268-e8d8-4506-8581-5c5ae5fc9ddf","Type":"ContainerDied","Data":"62a9eb50f75b1642d0e7c3d3ac22c8756c8e131756567dadbc60ad120b144457"}
Oct 30 19:28:11.504494 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.504326    4372 scope.go:122] "RemoveContainer" containerID="4ca7ffd27e66e78dcd9c9b64a3e4871bd7d940c8e3cc1a55324ae431bc52eaa9"
Oct 30 19:28:11.513199 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.513183    4372 scope.go:122] "RemoveContainer" containerID="4ca7ffd27e66e78dcd9c9b64a3e4871bd7d940c8e3cc1a55324ae431bc52eaa9"
Oct 30 19:28:11.513434 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:28:11.513418    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca7ffd27e66e78dcd9c9b64a3e4871bd7d940c8e3cc1a55324ae431bc52eaa9\": container with ID starting with 4ca7ffd27e66e78dcd9c9b64a3e4871bd7d940c8e3cc1a55324ae431bc52eaa9 not found: ID does not exist" containerID="4ca7ffd27e66e78dcd9c9b64a3e4871bd7d940c8e3cc1a55324ae431bc52eaa9"
Oct 30 19:28:11.513490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.513441    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca7ffd27e66e78dcd9c9b64a3e4871bd7d940c8e3cc1a55324ae431bc52eaa9"} err="failed to get container status \"4ca7ffd27e66e78dcd9c9b64a3e4871bd7d940c8e3cc1a55324ae431bc52eaa9\": rpc error: code = NotFound desc = could not find container \"4ca7ffd27e66e78dcd9c9b64a3e4871bd7d940c8e3cc1a55324ae431bc52eaa9\": container with ID starting with 4ca7ffd27e66e78dcd9c9b64a3e4871bd7d940c8e3cc1a55324ae431bc52eaa9 not found: ID does not exist"
Oct 30 19:28:11.528103 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.528078    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"]
Oct 30 19:28:11.535716 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.532482    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8a92cc-694d7878cd-jgw6t"]
Oct 30 19:28:11.593824 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.593797    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xp5z\" (UniqueName: \"kubernetes.io/projected/23924268-e8d8-4506-8581-5c5ae5fc9ddf-kube-api-access-2xp5z\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:11.593824 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:11.593822    4372 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23924268-e8d8-4506-8581-5c5ae5fc9ddf-plugins-config-volume\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:12.118230 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:12.118200    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qz7ks_641f9bfc-534b-4a1b-a4de-e68a140ee1f6/limitador/0.log"
Oct 30 19:28:12.263172 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:12.263120    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23924268-e8d8-4506-8581-5c5ae5fc9ddf" path="/var/lib/kubelet/pods/23924268-e8d8-4506-8581-5c5ae5fc9ddf/volumes"
Oct 30 19:28:27.696411 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:28:27.696361    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:28:30.035118 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:30.035074    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6c665b8798-czqkv_fd1bf3b0-ef42-4659-9d4e-5e7dbc3e1c31/maas-api/0.log"
Oct 30 19:28:30.606255 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:30.606216    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4af4b4e7-9hhml_6c0b8fad-f840-43ec-b061-220e42012129/revoke-keys/0.log"
Oct 30 19:28:34.354313 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:34.354275    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qz7ks_641f9bfc-534b-4a1b-a4de-e68a140ee1f6/limitador/0.log"
Oct 30 19:28:35.497578 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:35.497546    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bkpw4_02723f85-0a69-43c7-8482-c2cd80959ee3/discovery/0.log"
Oct 30 19:28:36.198916 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.198882    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"]
Oct 30 19:28:36.199102 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.199079    4372 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4" podUID="02723f85-0a69-43c7-8482-c2cd80959ee3" containerName="discovery" containerID="cri-o://c5146c784feae6ab6a1c4ba09c9de080c7f78141ca1da7ed7a0f800bfa67c49d" gracePeriod=30
Oct 30 19:28:36.255343 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.255314    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-998xh_ccc2f38a-84e5-43c3-98d3-0f39ea658a2e/istio-proxy/0.log"
Oct 30 19:28:36.447673 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.447639    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 19:28:36.496234 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.496173    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02723f85-0a69-43c7-8482-c2cd80959ee3-kube-api-access-tsv2s\" (UniqueName: \"kubernetes.io/projected/02723f85-0a69-43c7-8482-c2cd80959ee3-kube-api-access-tsv2s\") pod \"02723f85-0a69-43c7-8482-c2cd80959ee3\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") "
Oct 30 19:28:36.496371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.496241    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-csr-ca-configmap\") pod \"02723f85-0a69-43c7-8482-c2cd80959ee3\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") "
Oct 30 19:28:36.496371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.496266    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-csr-dns-cert\") pod \"02723f85-0a69-43c7-8482-c2cd80959ee3\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") "
Oct 30 19:28:36.496371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.496293    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-kubeconfig\") pod \"02723f85-0a69-43c7-8482-c2cd80959ee3\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") "
Oct 30 19:28:36.496371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.496316    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-cacerts\" (UniqueName: \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-cacerts\") pod \"02723f85-0a69-43c7-8482-c2cd80959ee3\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") "
Oct 30 19:28:36.496371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.496365    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02723f85-0a69-43c7-8482-c2cd80959ee3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/02723f85-0a69-43c7-8482-c2cd80959ee3-local-certs\") pod \"02723f85-0a69-43c7-8482-c2cd80959ee3\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") "
Oct 30 19:28:36.496679 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.496391    4372 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-token\" (UniqueName: \"kubernetes.io/projected/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-token\") pod \"02723f85-0a69-43c7-8482-c2cd80959ee3\" (UID: \"02723f85-0a69-43c7-8482-c2cd80959ee3\") "
Oct 30 19:28:36.496754 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.496692    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-csr-ca-configmap" pod "02723f85-0a69-43c7-8482-c2cd80959ee3" (UID: "02723f85-0a69-43c7-8482-c2cd80959ee3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:36.498773 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.498734    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02723f85-0a69-43c7-8482-c2cd80959ee3-kube-api-access-tsv2s" pod "02723f85-0a69-43c7-8482-c2cd80959ee3" (UID: "02723f85-0a69-43c7-8482-c2cd80959ee3"). InnerVolumeSpecName "kube-api-access-tsv2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:36.499314 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.499212    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-kubeconfig" pod "02723f85-0a69-43c7-8482-c2cd80959ee3" (UID: "02723f85-0a69-43c7-8482-c2cd80959ee3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:36.499387 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.499316    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-token" pod "02723f85-0a69-43c7-8482-c2cd80959ee3" (UID: "02723f85-0a69-43c7-8482-c2cd80959ee3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:36.499387 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.499319    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-csr-dns-cert" pod "02723f85-0a69-43c7-8482-c2cd80959ee3" (UID: "02723f85-0a69-43c7-8482-c2cd80959ee3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:36.499387 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.499337    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-cacerts" pod "02723f85-0a69-43c7-8482-c2cd80959ee3" (UID: "02723f85-0a69-43c7-8482-c2cd80959ee3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:36.499780 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.499758    4372 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02723f85-0a69-43c7-8482-c2cd80959ee3-local-certs" pod "02723f85-0a69-43c7-8482-c2cd80959ee3" (UID: "02723f85-0a69-43c7-8482-c2cd80959ee3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:36.597637 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.597606    4372 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-csr-ca-configmap\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.597637 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.597636    4372 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-csr-dns-cert\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.597800 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.597652    4372 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-kubeconfig\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.597800 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.597663    4372 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/02723f85-0a69-43c7-8482-c2cd80959ee3-cacerts\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.597800 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.597675    4372 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/02723f85-0a69-43c7-8482-c2cd80959ee3-local-certs\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.597800 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.597688    4372 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02723f85-0a69-43c7-8482-c2cd80959ee3-istio-token\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.597800 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.597701    4372 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsv2s\" (UniqueName: \"kubernetes.io/projected/02723f85-0a69-43c7-8482-c2cd80959ee3-kube-api-access-tsv2s\") on node \"ip-10-0-142-153.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:36.617373 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.617349    4372 generic.go:356] "Generic (PLEG): container finished" podID="02723f85-0a69-43c7-8482-c2cd80959ee3" containerID="c5146c784feae6ab6a1c4ba09c9de080c7f78141ca1da7ed7a0f800bfa67c49d" exitCode=0
Oct 30 19:28:36.617480 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.617395    4372 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"
Oct 30 19:28:36.617480 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.617402    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4" event={"ID":"02723f85-0a69-43c7-8482-c2cd80959ee3","Type":"ContainerDied","Data":"c5146c784feae6ab6a1c4ba09c9de080c7f78141ca1da7ed7a0f800bfa67c49d"}
Oct 30 19:28:36.617480 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.617452    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4" event={"ID":"02723f85-0a69-43c7-8482-c2cd80959ee3","Type":"ContainerDied","Data":"6c9774ea7724f5cab3b3e6a6ce0697b85d06e0d63f4d85d59780856c9b81389d"}
Oct 30 19:28:36.617657 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.617478    4372 scope.go:122] "RemoveContainer" containerID="c5146c784feae6ab6a1c4ba09c9de080c7f78141ca1da7ed7a0f800bfa67c49d"
Oct 30 19:28:36.633771 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.633745    4372 scope.go:122] "RemoveContainer" containerID="c5146c784feae6ab6a1c4ba09c9de080c7f78141ca1da7ed7a0f800bfa67c49d"
Oct 30 19:28:36.634032 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:28:36.634013    4372 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5146c784feae6ab6a1c4ba09c9de080c7f78141ca1da7ed7a0f800bfa67c49d\": container with ID starting with c5146c784feae6ab6a1c4ba09c9de080c7f78141ca1da7ed7a0f800bfa67c49d not found: ID does not exist" containerID="c5146c784feae6ab6a1c4ba09c9de080c7f78141ca1da7ed7a0f800bfa67c49d"
Oct 30 19:28:36.634105 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.634045    4372 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5146c784feae6ab6a1c4ba09c9de080c7f78141ca1da7ed7a0f800bfa67c49d"} err="failed to get container status \"c5146c784feae6ab6a1c4ba09c9de080c7f78141ca1da7ed7a0f800bfa67c49d\": rpc error: code = NotFound desc = could not find container \"c5146c784feae6ab6a1c4ba09c9de080c7f78141ca1da7ed7a0f800bfa67c49d\": container with ID starting with c5146c784feae6ab6a1c4ba09c9de080c7f78141ca1da7ed7a0f800bfa67c49d not found: ID does not exist"
Oct 30 19:28:36.640960 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.640940    4372 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"]
Oct 30 19:28:36.647921 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.647897    4372 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkpw4"]
Oct 30 19:28:36.980029 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:36.980008    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-qstgx_ce1f4775-99b1-4896-b58f-348c46e3b5f4/istio-proxy/0.log"
Oct 30 19:28:37.514845 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:37.514816    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-m4bn2_ac73b3bf-215f-43f2-afa9-b5155af04781/payload-pre-processing/0.log"
Oct 30 19:28:38.264143 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:38.264094    4372 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02723f85-0a69-43c7-8482-c2cd80959ee3" path="/var/lib/kubelet/pods/02723f85-0a69-43c7-8482-c2cd80959ee3/volumes"
Oct 30 19:28:38.677748 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:38.677711    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-hftzp_74a03d36-6248-4cfd-be2f-9415834a3337/main/0.log"
Oct 30 19:28:39.228105 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:39.228067    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-jn2fc_45b82fd5-5bcc-40ae-b071-4e026abe1d07/main/0.log"
Oct 30 19:28:39.592556 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:39.592527    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-9mxbq_0316c797-de93-4782-9506-7057fb3aa4a1/main/0.log"
Oct 30 19:28:39.771370 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:39.771324    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-c8xgm_0a7aab0a-df8b-486f-a479-bab120d33d03/main/0.log"
Oct 30 19:28:50.773374 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.773339    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xxpq7/must-gather-6hj5j"]
Oct 30 19:28:50.773795 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.773780    4372 cpu_manager.go:414] "containerMap: removing container" podUID="6c0b8fad-f840-43ec-b061-220e42012129" containerName="revoke-keys"
Oct 30 19:28:50.773843 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.773798    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c0b8fad-f840-43ec-b061-220e42012129" containerName="revoke-keys"
Oct 30 19:28:50.773843 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.773815    4372 cpu_manager.go:414] "containerMap: removing container" podUID="02723f85-0a69-43c7-8482-c2cd80959ee3" containerName="discovery"
Oct 30 19:28:50.773843 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.773823    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02723f85-0a69-43c7-8482-c2cd80959ee3" containerName="discovery"
Oct 30 19:28:50.773945 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.773850    4372 cpu_manager.go:414] "containerMap: removing container" podUID="23924268-e8d8-4506-8581-5c5ae5fc9ddf" containerName="payload-processing"
Oct 30 19:28:50.773945 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.773859    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23924268-e8d8-4506-8581-5c5ae5fc9ddf" containerName="payload-processing"
Oct 30 19:28:50.773945 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.773879    4372 cpu_manager.go:414] "containerMap: removing container" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main"
Oct 30 19:28:50.773945 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.773886    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main"
Oct 30 19:28:50.773945 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.773900    4372 cpu_manager.go:414] "containerMap: removing container" podUID="dd868454-02ed-4227-ae09-be713e3473d1" containerName="istio-proxy"
Oct 30 19:28:50.773945 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.773907    4372 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd868454-02ed-4227-ae09-be713e3473d1" containerName="istio-proxy"
Oct 30 19:28:50.774150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.773976    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="02723f85-0a69-43c7-8482-c2cd80959ee3" containerName="discovery"
Oct 30 19:28:50.774150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.773992    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="23924268-e8d8-4506-8581-5c5ae5fc9ddf" containerName="payload-processing"
Oct 30 19:28:50.774150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.774003    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="72834257-61fb-4596-8b82-5917990f7469" containerName="istio-proxy"
Oct 30 19:28:50.774150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.774014    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="22579f9b-ea36-4a4b-867b-87019a0b62c6" containerName="main"
Oct 30 19:28:50.774150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.774025    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd868454-02ed-4227-ae09-be713e3473d1" containerName="istio-proxy"
Oct 30 19:28:50.774150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.774035    4372 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c0b8fad-f840-43ec-b061-220e42012129" containerName="revoke-keys"
Oct 30 19:28:50.777490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.777471    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xxpq7/must-gather-6hj5j"
Oct 30 19:28:50.792230 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.792204    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xxpq7/must-gather-6hj5j"]
Oct 30 19:28:50.792358 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.792247    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xxpq7\"/\"default-dockercfg-8wkgl\""
Oct 30 19:28:50.793002 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.792976    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xxpq7\"/\"openshift-service-ca.crt\""
Oct 30 19:28:50.793314 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.793290    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xxpq7\"/\"kube-root-ca.crt\""
Oct 30 19:28:50.867402 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.867374    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nwzbq/must-gather-65gzr"]
Oct 30 19:28:50.870706 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.870682    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nwzbq/must-gather-65gzr"
Oct 30 19:28:50.872237 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.872213    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mz7g4/must-gather-nc7gz"]
Oct 30 19:28:50.875463 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.875443    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mz7g4/must-gather-nc7gz"
Oct 30 19:28:50.883272 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.883250    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nwzbq/must-gather-65gzr"]
Oct 30 19:28:50.892075 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.892050    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mz7g4/must-gather-nc7gz"]
Oct 30 19:28:50.892475 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.892444    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mz7g4\"/\"kube-root-ca.crt\""
Oct 30 19:28:50.892666 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.892616    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mz7g4\"/\"default-dockercfg-9jgvv\""
Oct 30 19:28:50.892666 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.892628    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nwzbq\"/\"openshift-service-ca.crt\""
Oct 30 19:28:50.892666 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.892626    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nwzbq\"/\"kube-root-ca.crt\""
Oct 30 19:28:50.892666 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.892646    4372 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mz7g4\"/\"openshift-service-ca.crt\""
Oct 30 19:28:50.892868 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.892731    4372 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nwzbq\"/\"default-dockercfg-jgvfw\""
Oct 30 19:28:50.912895 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.912870    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9566965e-f342-4077-9db4-28f67443723b-must-gather-output\") pod \"must-gather-6hj5j\" (UID: \"9566965e-f342-4077-9db4-28f67443723b\") " pod="openshift-must-gather-xxpq7/must-gather-6hj5j"
Oct 30 19:28:50.913482 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:50.913452    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7pb\" (UniqueName: \"kubernetes.io/projected/9566965e-f342-4077-9db4-28f67443723b-kube-api-access-gv7pb\") pod \"must-gather-6hj5j\" (UID: \"9566965e-f342-4077-9db4-28f67443723b\") " pod="openshift-must-gather-xxpq7/must-gather-6hj5j"
Oct 30 19:28:51.014562 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.014535    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9566965e-f342-4077-9db4-28f67443723b-must-gather-output\") pod \"must-gather-6hj5j\" (UID: \"9566965e-f342-4077-9db4-28f67443723b\") " pod="openshift-must-gather-xxpq7/must-gather-6hj5j"
Oct 30 19:28:51.014692 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.014575    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7pb\" (UniqueName: \"kubernetes.io/projected/9566965e-f342-4077-9db4-28f67443723b-kube-api-access-gv7pb\") pod \"must-gather-6hj5j\" (UID: \"9566965e-f342-4077-9db4-28f67443723b\") " pod="openshift-must-gather-xxpq7/must-gather-6hj5j"
Oct 30 19:28:51.014692 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.014599    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c7f5baf6-fe83-4ea2-9371-ab05a7624c7e-must-gather-output\") pod \"must-gather-nc7gz\" (UID: \"c7f5baf6-fe83-4ea2-9371-ab05a7624c7e\") " pod="openshift-must-gather-mz7g4/must-gather-nc7gz"
Oct 30 19:28:51.014843 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.014720    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcgl9\" (UniqueName: \"kubernetes.io/projected/45e19c9f-6aaf-4e96-a178-553f2a34ad0d-kube-api-access-wcgl9\") pod \"must-gather-65gzr\" (UID: \"45e19c9f-6aaf-4e96-a178-553f2a34ad0d\") " pod="openshift-must-gather-nwzbq/must-gather-65gzr"
Oct 30 19:28:51.014843 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.014760    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/45e19c9f-6aaf-4e96-a178-553f2a34ad0d-must-gather-output\") pod \"must-gather-65gzr\" (UID: \"45e19c9f-6aaf-4e96-a178-553f2a34ad0d\") " pod="openshift-must-gather-nwzbq/must-gather-65gzr"
Oct 30 19:28:51.014965 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.014854    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ng2r\" (UniqueName: \"kubernetes.io/projected/c7f5baf6-fe83-4ea2-9371-ab05a7624c7e-kube-api-access-5ng2r\") pod \"must-gather-nc7gz\" (UID: \"c7f5baf6-fe83-4ea2-9371-ab05a7624c7e\") " pod="openshift-must-gather-mz7g4/must-gather-nc7gz"
Oct 30 19:28:51.015017 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.014956    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9566965e-f342-4077-9db4-28f67443723b-must-gather-output\") pod \"must-gather-6hj5j\" (UID: \"9566965e-f342-4077-9db4-28f67443723b\") " pod="openshift-must-gather-xxpq7/must-gather-6hj5j"
Oct 30 19:28:51.029193 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.029119    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7pb\" (UniqueName: \"kubernetes.io/projected/9566965e-f342-4077-9db4-28f67443723b-kube-api-access-gv7pb\") pod \"must-gather-6hj5j\" (UID: \"9566965e-f342-4077-9db4-28f67443723b\") " pod="openshift-must-gather-xxpq7/must-gather-6hj5j"
Oct 30 19:28:51.091279 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.091256    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xxpq7/must-gather-6hj5j"
Oct 30 19:28:51.115869 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.115848    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c7f5baf6-fe83-4ea2-9371-ab05a7624c7e-must-gather-output\") pod \"must-gather-nc7gz\" (UID: \"c7f5baf6-fe83-4ea2-9371-ab05a7624c7e\") " pod="openshift-must-gather-mz7g4/must-gather-nc7gz"
Oct 30 19:28:51.115967 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.115876    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcgl9\" (UniqueName: \"kubernetes.io/projected/45e19c9f-6aaf-4e96-a178-553f2a34ad0d-kube-api-access-wcgl9\") pod \"must-gather-65gzr\" (UID: \"45e19c9f-6aaf-4e96-a178-553f2a34ad0d\") " pod="openshift-must-gather-nwzbq/must-gather-65gzr"
Oct 30 19:28:51.115967 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.115898    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/45e19c9f-6aaf-4e96-a178-553f2a34ad0d-must-gather-output\") pod \"must-gather-65gzr\" (UID: \"45e19c9f-6aaf-4e96-a178-553f2a34ad0d\") " pod="openshift-must-gather-nwzbq/must-gather-65gzr"
Oct 30 19:28:51.115967 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.115925    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ng2r\" (UniqueName: \"kubernetes.io/projected/c7f5baf6-fe83-4ea2-9371-ab05a7624c7e-kube-api-access-5ng2r\") pod \"must-gather-nc7gz\" (UID: \"c7f5baf6-fe83-4ea2-9371-ab05a7624c7e\") " pod="openshift-must-gather-mz7g4/must-gather-nc7gz"
Oct 30 19:28:51.116269 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.116247    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c7f5baf6-fe83-4ea2-9371-ab05a7624c7e-must-gather-output\") pod \"must-gather-nc7gz\" (UID: \"c7f5baf6-fe83-4ea2-9371-ab05a7624c7e\") " pod="openshift-must-gather-mz7g4/must-gather-nc7gz"
Oct 30 19:28:51.116336 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.116274    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/45e19c9f-6aaf-4e96-a178-553f2a34ad0d-must-gather-output\") pod \"must-gather-65gzr\" (UID: \"45e19c9f-6aaf-4e96-a178-553f2a34ad0d\") " pod="openshift-must-gather-nwzbq/must-gather-65gzr"
Oct 30 19:28:51.143736 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.141372    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ng2r\" (UniqueName: \"kubernetes.io/projected/c7f5baf6-fe83-4ea2-9371-ab05a7624c7e-kube-api-access-5ng2r\") pod \"must-gather-nc7gz\" (UID: \"c7f5baf6-fe83-4ea2-9371-ab05a7624c7e\") " pod="openshift-must-gather-mz7g4/must-gather-nc7gz"
Oct 30 19:28:51.143736 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.142943    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcgl9\" (UniqueName: \"kubernetes.io/projected/45e19c9f-6aaf-4e96-a178-553f2a34ad0d-kube-api-access-wcgl9\") pod \"must-gather-65gzr\" (UID: \"45e19c9f-6aaf-4e96-a178-553f2a34ad0d\") " pod="openshift-must-gather-nwzbq/must-gather-65gzr"
Oct 30 19:28:51.187411 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.187386    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nwzbq/must-gather-65gzr"
Oct 30 19:28:51.192504 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.192482    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mz7g4/must-gather-nc7gz"
Oct 30 19:28:51.375047 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.375019    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xxpq7/must-gather-6hj5j"]
Oct 30 19:28:51.377037 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:28:51.377011    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9566965e_f342_4077_9db4_28f67443723b.slice/crio-9a9884d1f03817b52323bf622619a26062b00361d9b78b336283c623ab9bfe76 WatchSource:0}: Error finding container 9a9884d1f03817b52323bf622619a26062b00361d9b78b336283c623ab9bfe76: Status 404 returned error can't find the container with id 9a9884d1f03817b52323bf622619a26062b00361d9b78b336283c623ab9bfe76
Oct 30 19:28:51.378804 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.378778    4372 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:28:51.495554 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.495530    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nwzbq/must-gather-65gzr"]
Oct 30 19:28:51.496850 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:28:51.496824    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45e19c9f_6aaf_4e96_a178_553f2a34ad0d.slice/crio-4d7353d16a633a989044e64859f94fbccd27e2b70f7c17521bd0db7ef5bbfaf8 WatchSource:0}: Error finding container 4d7353d16a633a989044e64859f94fbccd27e2b70f7c17521bd0db7ef5bbfaf8: Status 404 returned error can't find the container with id 4d7353d16a633a989044e64859f94fbccd27e2b70f7c17521bd0db7ef5bbfaf8
Oct 30 19:28:51.507490 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.507468    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mz7g4/must-gather-nc7gz"]
Oct 30 19:28:51.508449 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:28:51.508429    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7f5baf6_fe83_4ea2_9371_ab05a7624c7e.slice/crio-c4a438d584bb22266602703da462cfe9f435b1b4223fd3b00ce0857aa6d297b4 WatchSource:0}: Error finding container c4a438d584bb22266602703da462cfe9f435b1b4223fd3b00ce0857aa6d297b4: Status 404 returned error can't find the container with id c4a438d584bb22266602703da462cfe9f435b1b4223fd3b00ce0857aa6d297b4
Oct 30 19:28:51.690096 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.690065    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mz7g4/must-gather-nc7gz" event={"ID":"c7f5baf6-fe83-4ea2-9371-ab05a7624c7e","Type":"ContainerStarted","Data":"c4a438d584bb22266602703da462cfe9f435b1b4223fd3b00ce0857aa6d297b4"}
Oct 30 19:28:51.691280 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.691256    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nwzbq/must-gather-65gzr" event={"ID":"45e19c9f-6aaf-4e96-a178-553f2a34ad0d","Type":"ContainerStarted","Data":"4d7353d16a633a989044e64859f94fbccd27e2b70f7c17521bd0db7ef5bbfaf8"}
Oct 30 19:28:51.692318 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:51.692298    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xxpq7/must-gather-6hj5j" event={"ID":"9566965e-f342-4077-9db4-28f67443723b","Type":"ContainerStarted","Data":"9a9884d1f03817b52323bf622619a26062b00361d9b78b336283c623ab9bfe76"}
Oct 30 19:28:52.701379 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:52.701331    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nwzbq/must-gather-65gzr" event={"ID":"45e19c9f-6aaf-4e96-a178-553f2a34ad0d","Type":"ContainerStarted","Data":"e3e9c382863f31275264f3e515cbbaa2ab14961b7583a281f8f03642e2fa82ea"}
Oct 30 19:28:52.701379 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:52.701379    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nwzbq/must-gather-65gzr" event={"ID":"45e19c9f-6aaf-4e96-a178-553f2a34ad0d","Type":"ContainerStarted","Data":"eecf0965a0bffa58e07b30937628c58ae01cdc3999f43e5821711b82d9266ef1"}
Oct 30 19:28:52.725522 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:52.725474    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nwzbq/must-gather-65gzr" podStartSLOduration=1.923536608 podStartE2EDuration="2.725460275s" totalImagesPullingTime="801.923667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:52.722019616 +0000 UTC m=+3346.590437949" watchObservedRunningTime="2026-10-30 19:28:52.725460275 +0000 UTC m=+3346.593878605"
Oct 30 19:28:55.077912 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:55.077880    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qmbfz_faeb3114-7d5c-4d94-a00c-7af6f1c175df/global-pull-secret-syncer/0.log"
Oct 30 19:28:55.399959 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:55.399842    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8vlwv_41efb608-1c74-4ab4-9693-53495bfc26b9/konnectivity-agent/0.log"
Oct 30 19:28:55.557589 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:28:55.557552    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-153.us-east-2.compute.internal_c870f7ea63961faf808ec834ef3f426f/haproxy/0.log"
Oct 30 19:28:57.781011 ip-10-0-142-153 kubenswrapper[4372]: E1030 19:28:57.780971    4372 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2t9nh" containerName="extractor"
Oct 30 19:29:03.670904 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:03.668680    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qz7ks_641f9bfc-534b-4a1b-a4de-e68a140ee1f6/limitador/0.log"
Oct 30 19:29:06.181305 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:06.181270    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sbdjx_bd3ac4bf-016e-496c-9895-c9fd1d60d897/node-exporter/0.log"
Oct 30 19:29:06.224150 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:06.222093    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sbdjx_bd3ac4bf-016e-496c-9895-c9fd1d60d897/kube-rbac-proxy/0.log"
Oct 30 19:29:06.267823 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:06.267720    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sbdjx_bd3ac4bf-016e-496c-9895-c9fd1d60d897/init-textfile/0.log"
Oct 30 19:29:06.670568 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:06.670534    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-kk6bc_c80b2cb2-7c3f-42dd-ba8f-017131416961/prometheus-operator/0.log"
Oct 30 19:29:06.694840 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:06.694809    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-kk6bc_c80b2cb2-7c3f-42dd-ba8f-017131416961/kube-rbac-proxy/0.log"
Oct 30 19:29:06.729206 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:06.729152    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-h68bb_e07c1ad1-f5e7-49b0-b7b7-078cb19aa615/prometheus-operator-admission-webhook/0.log"
Oct 30 19:29:07.088713 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.086978    4372 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"]
Oct 30 19:29:07.249027 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.248607    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.249603 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.249579    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"]
Oct 30 19:29:07.307371 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.307338    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0-proc\") pod \"perf-node-gather-daemonset-vbpr9\" (UID: \"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.307566 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.307382    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hn5z\" (UniqueName: \"kubernetes.io/projected/6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0-kube-api-access-7hn5z\") pod \"perf-node-gather-daemonset-vbpr9\" (UID: \"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.307566 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.307410    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0-lib-modules\") pod \"perf-node-gather-daemonset-vbpr9\" (UID: \"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.307566 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.307438    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0-sys\") pod \"perf-node-gather-daemonset-vbpr9\" (UID: \"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.307566 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.307460    4372 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0-podres\") pod \"perf-node-gather-daemonset-vbpr9\" (UID: \"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.411688 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.411612    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0-proc\") pod \"perf-node-gather-daemonset-vbpr9\" (UID: \"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.411688 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.411651    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hn5z\" (UniqueName: \"kubernetes.io/projected/6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0-kube-api-access-7hn5z\") pod \"perf-node-gather-daemonset-vbpr9\" (UID: \"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.411688 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.411675    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0-lib-modules\") pod \"perf-node-gather-daemonset-vbpr9\" (UID: \"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.411948 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.411760    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0-proc\") pod \"perf-node-gather-daemonset-vbpr9\" (UID: \"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.411948 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.411772    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0-lib-modules\") pod \"perf-node-gather-daemonset-vbpr9\" (UID: \"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.411948 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.411815    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0-sys\") pod \"perf-node-gather-daemonset-vbpr9\" (UID: \"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.411948 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.411858    4372 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0-podres\") pod \"perf-node-gather-daemonset-vbpr9\" (UID: \"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.412165 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.411961    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0-sys\") pod \"perf-node-gather-daemonset-vbpr9\" (UID: \"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.412165 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.412037    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0-podres\") pod \"perf-node-gather-daemonset-vbpr9\" (UID: \"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.425324 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.425303    4372 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hn5z\" (UniqueName: \"kubernetes.io/projected/6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0-kube-api-access-7hn5z\") pod \"perf-node-gather-daemonset-vbpr9\" (UID: \"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.641267 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.640943    4372 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:07.987186 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:07.984014    4372 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"]
Oct 30 19:29:07.998684 ip-10-0-142-153 kubenswrapper[4372]: W1030 19:29:07.997568    4372 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6dfd8d81_bc5e_4c13_aa5c_c4a1bf6759f0.slice/crio-2569bd0e49f717d1ba67dfa27752bceeb3e916c36d3bd462b5e9b510e5dc9f21 WatchSource:0}: Error finding container 2569bd0e49f717d1ba67dfa27752bceeb3e916c36d3bd462b5e9b510e5dc9f21: Status 404 returned error can't find the container with id 2569bd0e49f717d1ba67dfa27752bceeb3e916c36d3bd462b5e9b510e5dc9f21
Oct 30 19:29:08.981656 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:08.981614    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9" event={"ID":"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0","Type":"ContainerStarted","Data":"ee2fbdd6c7af3ccfdf5d3fd443aaaf7f20acaa479122eff9b5f1b2a699a8de95"}
Oct 30 19:29:08.982230 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:08.981684    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9" event={"ID":"6dfd8d81-bc5e-4c13-aa5c-c4a1bf6759f0","Type":"ContainerStarted","Data":"2569bd0e49f717d1ba67dfa27752bceeb3e916c36d3bd462b5e9b510e5dc9f21"}
Oct 30 19:29:08.982464 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:08.982394    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:09.002917 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:09.002855    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9" podStartSLOduration=2.002841087 podStartE2EDuration="2.002841087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:09.002667672 +0000 UTC m=+3362.871086005" watchObservedRunningTime="2026-10-30 19:29:09.002841087 +0000 UTC m=+3362.871259416"
Oct 30 19:29:09.994248 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:09.994146    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mz7g4/must-gather-nc7gz" event={"ID":"c7f5baf6-fe83-4ea2-9371-ab05a7624c7e","Type":"ContainerStarted","Data":"0fa00983aa3aa56c9d70ccbaf0e43e5ffdedaed7ed6a5af54e995f3eab7a466f"}
Oct 30 19:29:09.994248 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:09.994206    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mz7g4/must-gather-nc7gz" event={"ID":"c7f5baf6-fe83-4ea2-9371-ab05a7624c7e","Type":"ContainerStarted","Data":"54bb821d459a5b3ff33a1c1039eaae73fd1e786cc064581ce86b5990dced7d36"}
Oct 30 19:29:10.013578 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:10.013511    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mz7g4/must-gather-nc7gz" podStartSLOduration=2.167486519 podStartE2EDuration="20.013494536s" totalImagesPullingTime="17.846008017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:10.013190087 +0000 UTC m=+3363.881608416" watchObservedRunningTime="2026-10-30 19:29:10.013494536 +0000 UTC m=+3363.881912866"
Oct 30 19:29:10.359618 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:10.359589    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-577c7d6796-m7ztx_9f93f0a1-d934-45f5-9397-98c057e4f050/console/0.log"
Oct 30 19:29:10.419141 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:10.419089    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-fx7q7_d02d01c9-1b8c-4632-866c-3fecfda75c4a/download-server/0.log"
Oct 30 19:29:11.008328 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:11.008198    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xxpq7/must-gather-6hj5j" event={"ID":"9566965e-f342-4077-9db4-28f67443723b","Type":"ContainerStarted","Data":"e4cbd6012bf72bce2acf3b3fab637171c25902f538ee5182bfa40afefe470ef2"}
Oct 30 19:29:11.008328 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:11.008262    4372 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xxpq7/must-gather-6hj5j" event={"ID":"9566965e-f342-4077-9db4-28f67443723b","Type":"ContainerStarted","Data":"c66fd99d98124f61919237809804f5ecc141faf33a9a63dd83ba43df84fc6486"}
Oct 30 19:29:11.030010 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:11.029946    4372 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xxpq7/must-gather-6hj5j" podStartSLOduration=2.04865339 podStartE2EDuration="21.02992679s" totalImagesPullingTime="18.9812734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:11.028960832 +0000 UTC m=+3364.897379163" watchObservedRunningTime="2026-10-30 19:29:11.02992679 +0000 UTC m=+3364.898345120"
Oct 30 19:29:13.049324 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:13.049290    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2j794_74d61d4b-1a8f-4256-aae6-c51763689c92/dns/0.log"
Oct 30 19:29:13.081008 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:13.080957    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2j794_74d61d4b-1a8f-4256-aae6-c51763689c92/kube-rbac-proxy/0.log"
Oct 30 19:29:13.335603 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:13.335574    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h64hp_9c151e21-28d5-4d4a-8b95-b92d623582c5/dns-node-resolver/0.log"
Oct 30 19:29:14.255819 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:14.255789    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-fd56c4b66-72fh4_fa15387d-68c0-4a12-a921-313ba8743b0a/registry/0.log"
Oct 30 19:29:14.286370 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:14.286344    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9mqb5_382d4192-0bfc-4626-820d-5cf183296180/node-ca/0.log"
Oct 30 19:29:15.012777 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:15.012747    4372 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-vbpr9"
Oct 30 19:29:17.892246 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:17.891969    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-998xh_ccc2f38a-84e5-43c3-98d3-0f39ea658a2e/istio-proxy/0.log"
Oct 30 19:29:18.042627 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:18.042588    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-qstgx_ce1f4775-99b1-4896-b58f-348c46e3b5f4/istio-proxy/0.log"
Oct 30 19:29:18.165645 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:18.165546    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-m4bn2_ac73b3bf-215f-43f2-afa9-b5155af04781/payload-pre-processing/0.log"
Oct 30 19:29:21.253966 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:21.253929    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vgjw8_dae5359a-4e6c-4a5d-a688-aff0f02cb240/serve-healthcheck-canary/0.log"
Oct 30 19:29:21.781515 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:21.781482    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-998xh_ccc2f38a-84e5-43c3-98d3-0f39ea658a2e/istio-proxy/0.log"
Oct 30 19:29:21.881401 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:21.881369    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-qstgx_ce1f4775-99b1-4896-b58f-348c46e3b5f4/istio-proxy/0.log"
Oct 30 19:29:22.005408 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:22.005371    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2t9nh_76637ad2-0d7d-4c03-a808-13e0125424a2/kube-rbac-proxy/0.log"
Oct 30 19:29:22.019431 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:22.019400    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-m4bn2_ac73b3bf-215f-43f2-afa9-b5155af04781/payload-pre-processing/0.log"
Oct 30 19:29:22.049073 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:22.048994    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2t9nh_76637ad2-0d7d-4c03-a808-13e0125424a2/exporter/0.log"
Oct 30 19:29:22.083069 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:22.083043    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2t9nh_76637ad2-0d7d-4c03-a808-13e0125424a2/extractor/0.log"
Oct 30 19:29:22.351049 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:22.350994    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-xn6pz-hnbfq_c57a61f9-b181-4622-8e40-68402dde03fc/insights-gathering/0.log"
Oct 30 19:29:24.816603 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:24.816570    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-998xh_ccc2f38a-84e5-43c3-98d3-0f39ea658a2e/istio-proxy/0.log"
Oct 30 19:29:24.883821 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:24.883790    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-qstgx_ce1f4775-99b1-4896-b58f-348c46e3b5f4/istio-proxy/0.log"
Oct 30 19:29:24.969148 ip-10-0-142-153 kubenswrapper[4372]: I1030 19:29:24.965904    4372 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-m4bn2_ac73b3bf-215f-43f2-afa9-b5155af04781/payload-pre-processing/0.log"